Databases and Information Systems: ChatGPT in DBIS Teaching

Currently there are many discussions about ChatGPT and teaching (and exams). For the general strategy of the university see here (in German). Some of the references given there are useful:

• TU München: ChatGPT - Didaktische Perspektiven. Open the first item and follow the link Handreichung ChatGPT (Deutsch) bzw. (Englisch). "Verhalte dich wie ein Software-Entwickler. Schreibe ein Programm ... ".
• A report by scientists who work in related fields for long time that describes how ChatGPT works and that gives guidance for users, like
  "Act as a professional spelling and grammar corrector and improver" - wait for response, paste text.
  "Please revise the following sentence to make it clearer and more concise: ..."
• Juristische Gedanken (in German) zu ChatGTP im bayrischen Prüfungsrecht. Konträr zu dem Vorgehen unserer Universität (die aber ja auch nicht in Bayern ist).

For our database teaching, we are also experimenting with ChatGPT. Some experiments shown below (April 2023) showed that it is neither powerful, nor are its answers reliable. It is not very helpful for work, but it is interesting to see how artificial intelligence struggles with rather simple tasks like SQL queries and relational algebra trees. There seems to be a mismatch problem between (i) "soft" deep-learning-based AI, like chatting, and (ii) concrete tasks like writing syntactically and structurally correct queries, or (iii) facts that could be looked up in a database (actually, often ChatGPT gives answers that contradict facts that it knows (if asked critically).

ChatGPT's Conflict between Data and Babbling

Dialogues with ChatGPT show that on many topics, it seems to have enough data available to give serious answers, but that instead its KI component prefers not to use data, but to invent "facts" and to give wrong, even absurd, answers. Never ever believe what ChatGPT says!

• Geography: strange ideas about central european rivers (8 min, in German, 26.4.2023)
  Highlights translated:
  "Although Switzerland is not directly crossed by the Danube, there are some rivers in Switzerland that flow into the Danube, such as the Iller, which flows through Lake Constance and then empties into the Danube."
  "The Rhine, on the other hand, is a major river in Europe and an important tributary of the Danube. The Rhine and the Danube are connected to each other through the so-called Rhine-Main-Danube Canal, which connects the Rhine to the Danube. Thus, the water of the Reuss ultimately flows into the Black Sea via the Aare, the Rhine, and the Danube."
• History of German Cars: (no recording)
  "After August Horch founded Audi Automobilwerke GmbH in Zwickau in 1909, the company later became part of Auto Union AG, which also included the brands DKW, Horch, and Wanderer. Auto Union AG was acquired by Volkswagenwerk AG in 1932, which is now known as the Volkswagen Group."
  (simply impossible: the VW AG was only "founded" by the Hitler Regime in 1937).
  "The Audi 60 was a model produced by the German car manufacturer Audi between 1968 and 1972. The Audi 60 was a mid-sized car and the first model marketed under the Audi name after Audi was taken over by Volkswagen in 1965. It had a four-cylinder gasoline engine and was equipped with rear-wheel drive. Later, models with all-wheel drive were also offered."
  (it had -like all DKW/Audi- front-wheel drive, and there was by far no Audi 4WD in the 1960s - the first Quattro was built in 1980).
  "The Audi 60 was produced between 1968 and 1972 and was the first Audi produced after its acquisition by Volkswagen. It was available with various engines, including four-cylinder and six-cylinder gasoline engines."
  (there was no Audi-6-cylinder until 1990, since this was excluded by the contract between VW and Mercedes from 1965 for 25 years).

A first session with our Mondial database and SQL (19.4.2023)

• Give ChatGPT the CREATE TABLE statements of the database, then it can (try to) create SQL queries for textually described questions.
• Recording (in german)
• Conjunctive Query: OK.
• Negation with subquery: the first attempt is usually wrong, using atomic negation. Given a counterexample, it excuses and gives the correct variant.
• GROUP BY: not tested in this session.
• relational division: OK.

How does ChatGPT perform in a written database exam (video in German) (22.4.2023)

In a session with ChatGPT, the WS2018/19 exam ("Elections (Landtagswahl) in Hessen") was fed to ChatGPT. With a lot of help and hints and generous grading, it might pass the exam in 100 minutes working time.

• Recording (in german)
• Text understanding: seems to be quite good.
• ER Diagram: it uses a nonstandard notation which is more UML Class diagrams than true ER. Relationships are only lines, thus only binary ones without attributes, UML-style cardinalities. Graphical notation is broken (obviously, it tried SVG), it returns a textual description of the entity types and relationships. It invents additional entity types that are not mentioned in the task (e.g. individual voters and their votes, that must obviously not be stored in anonymous elections).
• Transformation/generation of the relational model: returns CREATE TABLEs. It did not stick with the task carefully, and invented additional entity types. It is able to refine the model based on hints, but it does not always use the same tables in the following.
• SQL:
  • in many cases where different alternative modelings are possible, it deviates from the schema that it proposed before.
  • simple conjunctive queries: ok.
  • negation with subqueries: the first attempt is usually wrong, using atomic negation. Given a counterexample, it excuses and gives the correct variant.
  • GROUP BY: in this slightly nontrivial case, fails.
  • relational division: competent.
• Relational Algebra: fails. Even in simple cases, it produces strange answers, broken syntax, mixed with SQL and logical quantifiers. Absolutely strange that there is obviously no syntax check of the result (and it cannot have "learnt" this rubbish anywhere).
  This shows the problems of combining strict and safe algorithmic processes with black-box Deep Learning.
• SQL updates: text understanding is suprisingly strong (I did not expect that it understands what to do). Creates the correct statements and shows that it really "understood" the task in its answering comment.
• Theory exercise: Bluff! Mathematically, nonsense - in well-formulated arguments (Artificial "intelligence" at its best).
• text question on technical understanding: surprisingly competent.

• There are some fragments of the answers (in German).