I started getnextline (the 42 project) on 19th Dec 23, and finally passed the project yesterday on 25th Feb 2024. It has been quite a journey! At times, it felt like I was stuck beyond stuck… it felt like I fell into a tar pit AND quicksand at the same time: I felt like I was drowning, AND bothered by the sand in my underwear…

Now that I’ve came out of the tarpit/quicksand and just took a fresh shower, I think it’s useful to write down my own retrospective of the project, before I forget everything…

What happened

TLDR - I rewrote the function four times. Here are the photos to prove it:

21 Dec 23 -

  • My classmate Selvam had sat me down and explained the outline of the project to me. I’m super grateful that he took the time to sit me down and give me an overview!
  • I watched this Youtube video from Nikito (student at 42 Paris) to understand the problem better. It was a very good explanation about the problem, and (an) overall solution strategy.
  • I used the diagram above to try and breakdown my understanding of the problem into specific procedures & functions.
  • I also read the very well-written articles by Mia Combeau (another 42 Paris student), which explained file descriptors and the read function, and static variables.

2-4 Jan 24: At this point, my approach was to try and iterate off the working sample code for reading from an fd, rather than trying to architect everything: the build-something-simple-and-build-again-after-it-works approach had worked for the previous project, ft_printf (which required us to duplicate the printf function in C). I was able to iterate until I could pass an unedited string from the buffer (which was read from fd), into the stash, and then to the result variable, without editing the string. When that worked, I added functions to check & trim the string, but it got very confusing… but I was able to return a single line.

Except I had a whole ton of memory errors, and I couldn’t figure out where the memory leaks were, because my code was complete spaghetti code. I decided I needed to restart, since it was the new year (Jan 2). Talking to another classmate who was also working on gnl (shortform for get_next_line), he was also restarting his code. He also taught me how to fix the problem with the francinette testing tool for get_next_line, as it wasn’t working on our Ubuntu Linux computers (has to do with the compiler used: you need to change the compiler from clang++ to g++ for the gnl tests in the tool).

I also realised I needed a more architectural perspective of the project, so I did a proper flow-chart of the project:

Using the flowchart, my own main (with various txt files for my own testing), francinette tests, and also learning how to use valgrind and gdb, I was able to gradually rework the code with consulting my classmates (Sean, Avery and Jeff Lim) over three weeks, until it looked something like this:

Then from 21 Jan until 5 Feb, I was trying to refactor my code to meet the norminette requirements. I also had this very strange thing where my code would work with BUFFER_SIZE=1000000 (i.e. 1e6) but not 1e7, where the main function’s stack would overflow. I would never successfully refactor the code to meet 42’s norminette code format standard.

At this point, my friend Mr T had returned to 42, started gnl and finished it within a week or so (!) And he helped me look at my code, and gave a whole bunch of comments. Most pertinently, he had a couple of very useful comments:

  • my gnl function had too many return statements. This, I later realized, makes the code very brittle and not reusable.
  • at the conceptual level of the problem, he commented: “when do you really need the static variable?” I replied that it was needed in order to build the string. And he pointed out that actually, the result and buffer variables remain in the same scope throughout: you only really need the static variable to store the remainder of a returned line. And that remainder will always be equal or less than BUFFER_SIZE. So you can actually just build the string in the returned variable, rather than in the static variable. -lightbulb-
  • he also commented that my helper functions were constantly returning a heap-allocated variable, which meant that I constantly needed to track where the heap-allocated variable was freed. He instead shared how he had created void functions, passing pointers-to-pointers as arguments, and was able to malloc & free within the function: that removed a lot of the need to track the frees, because of the very clever function design.
  • he suggested rewriting from scratch, rather than trying to patch my existing code.

So on 5 Feb, I locked my old working code in another git branch, and started work on another design:

From 5 Feb to 19 Feb, it was an absolute grind… I was churning out function by function, and testing them before building up the full gnl. At one point, I realised that I didn’t really understand why and how the leaks were happening, and created a small sample code to create valgrind leaks.

On 19th Feb, Mr T helped explain (and showed through live demo how he solved the leaks, before removing his code) how memory leaks are often not where the valgrind errors are. He also pointed out that I was often not considering the reassignment after a 2nd or 3rd time execution of gnl. And he also commented that my understanding of memory was still not very strong.

So I went to dig up and studied this book, to clarify my understanding of memory

In my frustration with debugging memory leaks, I also came across this very cool article that showed how to link valgrind with gdb for debugging: this was (and still is) super useful, because it doesn’t just print the valgrind error; I can examine the variable values near the error area, and understand much more clearly what the program is doing around the error.

Both the clearer understanding of memory allocation, pointers & strings, and the use of vgdb allowed me to debug my code of all memory leaks and read/write errors by 22 Feb. I then spent 23 Feb refactoring the code, which was completed in one day (vs. the previous refactoring that took 3 weeks and was unsuccessful!!), and defended my submission on 25th Feb.

Whew! That was quite a journey!

What to continue doing

  • Understand and solve at the right abstraction level first: conceptual level, before more granular levels. Take time to write out the overall logic, before attempting to code. It doesn’t have to be perfect, but doing so correctly saves a lot of time.
  • Continue writing a main function in a separate main.c, and using that to compile and test the program. => this allowed me to have very tight build-test-debug-iterate loops, vs. relying solely on the francinette program. It also had the benefit of completely eliminating the error of creating non-compilable code.
  • Continue reading and studying on concepts that are unclear, and …
  • Writing small programs to understand the concept for oneself.
  • using debugging tools like valgrind, gdb and vgdb.
  • Using automated testers like francinette last, if at all.
  • Testing every function before adding/using them.
  • Writing out a** todo-list** in a program’s comments, and using that to track progress.

What to stop doing

  • Just iterating from the groundup without any overall plan: that worked for ft_printf, but has completely, completely failed as an approach for gnl.
  • Just reading Github code bases of other people. While it is useful to get ideas, it can be very easy to get overwhelmed, and also to just copy code without understanding it fully. I have found it much more useful to just talk to people in class rather than to just look at Github repos.

I hope this writeup is also useful for others, especially if you’ve been very stuck on gnl.

Useful Resources for gnl