The increasing digitization in Germany can be observed particularly in students’ learning environments. Large-scale assessments have also recently switched from paper-based to computer-based tests. In TIMSS 2019, this change was accompanied by a mode effect study. Two years before the main survey, 847 fourth-graders participated in this study and were randomly assigned to take either the paper-based or the computer-based achievement test in mathematics and science first. In this paper, we address the question to what extent students’ achievement depends on the survey mode. We firstly conduct a generalizability study to assess the amount of variance in item responses that is explained by mode effects. Secondly, we investigate how much the relations between difficulty-generating characteristics of the test items and students' achievement differ by mode. The results show that the assessment of students' skills in both domains is affected by small mode effects. In the computer-based tests, students’ achievement is estimated to be slightly lower (mathematics: OR = 0.87, p = .006, science: OR = 0.86, p = .006). In mathematics this holds true for items addressing data and problem-solving in particular. For science we cannot make a similar claim. In the paper, the relevance of these findings for large-scale assessments and educational practice in Germany is critically discussed.