Check out the story of Lablup

Discover how professionals are growing with Lablup

Taeeun Kim

In the summer of 2025, from June to August, I worked as a research intern at Lablup. Rather than building AI models directly, Lablup focuses on building and improving the infrastructure needed for AI research and service operation. Centered around Backend.AI, the company works on areas such as GPU resource management, experiment environment setup, workload execution, and automation tools.

Before the internship, most of my experience with AI had been close to model training and optimization. I was more familiar with choosing models, training them, and improving their performance. The work I took on at Lablup, however, sat one layer outside of that. I analyzed GPU benchmarks, used FastTrack to identify areas for improvement, built a PyCon booth event page, and submitted pull requests to an open-source project. I also worked with internal experiment environments, recorded issues, documented results, and organized findings so the team could reuse them later.

In this post, I would like to summarize what I worked on during the internship and what I learned from dealing not with models themselves, but with the environments that run and operate them.

## What I Worked On

My work as a research intern began with getting familiar with the Backend.AI ecosystem. I used WebUI and FastTrack, recorded points where issues appeared in realistic usage scenarios, and, when needed, reported issues or fixed them directly. At the same time, I worked on tasks closer to the research team's work, such as GPU benchmark analysis.

At first, the structure of Backend.AI and FastTrack felt unfamiliar. But as I ran workloads, changed configurations, and compared results, I began to see what matters in AI infrastructure. Model performance is important, but so are the stability of the execution environment, repeatable experiments, interpretable metrics, and tool usability.

The main projects I worked on during the internship fell into three areas:

* analyzing GPU benchmark results and improving performance metrics
* fixing bugs and usability issues I found while using FastTrack
* building a feedback collection page for the PyCon 2025 booth event

In addition to these tasks, documenting the process and results, discussing them with teammates, and making sure they could lead to follow-up work were also important parts of the internship. I started out mostly as a user of the product, but over time I became more involved in reproducing issues, narrowing down causes, suggesting fixes, and implementing changes directly.

## Benchmarking GPU Workloads

One of my main projects was GPU benchmark analysis. The goal was to evaluate how different configurations affected benchmark performance and to begin automating parts of the process through FastTrack. This was not just a matter of running benchmarks. I had to compare results across different settings, investigate why some results varied, and check stability through repeated runs.

Benchmarking AI workloads sounds straightforward at first: run a workload, collect numbers, and compare the results. In practice, the results can vary for reasons that are not always obvious. Network latency, run-to-run variance, request patterns, and system-level conditions can all affect the metrics. A single average value is often not enough to explain what is happening.

This led me to examine the consistency of benchmark results across multiple runs and to think more carefully about what should be measured. I experimented with reducing the impact of network latency and checked how stable the results remained when running the same configuration multiple times. In addition to throughput and latency, variance became an important signal. For inference workloads in particular, per-request token throughput variance can reveal instability that aggregate metrics may hide.

I documented these experiments and analyses in Lablup's internal Knowledge Base so the team could refer back to them later. Rather than only collecting results in tables, I also recorded which conditions produced which differences, which results needed further verification, and what should be considered when automating the process later. I then added per-request token throughput variance metrics to the research team's benchmarking script. It was a small contribution, but it made benchmark results easier to interpret beyond a single representative number.

The biggest lesson I took from this work is that benchmarking is not simply about producing faster numbers. What matters is making the measurement process itself trustworthy. For AI infrastructure, reliable benchmarks require repeatability, enough context to explain variance, and automation that reduces inconsistency from manual work.

## FastTrack Usability and Open-Source Contribution

While working on GPU benchmarks, I used FastTrack frequently. FastTrack helps users configure experiments and automation tasks more easily, and as I used it in practice, I naturally started paying attention not only to its features but also to its user flow and interface. Following the steps needed for benchmark automation made it easier to notice which inputs were confusing, which states needed to be clearer, and which behaviors felt different from what a user might expect.

When you use a tool directly, small frictions can feel larger than they look. A confusing label, a missing state, or an unclear workflow may seem minor in isolation. But when a user is trying to run an experiment or manage resources, these small points of friction can affect both speed and accuracy. In infrastructure software, usability is not separate from technical quality.

Early in the internship, I reported a small issue related to a label in a FastTrack modal. The issue was discussed and addressed quickly. The change itself was small, but it showed me how a problem found during actual use could turn into a product improvement.

![FastTrack interface](https://cdn.lablup.com/review_image_02eb776321.png)

As I continued using FastTrack, I found more bugs and usability improvements, and eventually began contributing directly to the project. During the internship, I submitted six pull requests to FastTrack, resolved bug reports, and participated in discussions about improving the user experience. At first, I mainly organized issues and asked questions. Gradually, I became able to read through the codebase, understand the scope of a change, and prepare fixes myself.

Through this process, I learned that developer tools improve through small, concrete iterations. Each issue report, code review, and pull request becomes part of a larger product feedback loop. This loop is especially important in open-source infrastructure projects, where the boundary between user, contributor, and maintainer can be fluid.

## Building a Feedback Collection Page for PyCon

As PyCon 2025 approached, I also worked on an event page for Lablup's booth. The page was designed to let participants compare two AI-generated blog outputs with one written by a human and submit their feedback. Rather than being a simple promotional page, it was closer to a small experiment tool: participants needed to read multiple outputs, make a selection, and have their responses stored in a form that could be analyzed later.

I built the page with Next.js, Tailwind CSS, and shadcn/ui. Google Apps Script handled data storage in Google Sheets, and the page was deployed on AWS Amplify. Because the work involved the frontend interface, response submission flow, Google Sheets integration, and deployment setup, even a relatively small application became a useful exercise in building a lightweight data collection workflow for a real event environment.

![PyCon booth feedback page](https://cdn.lablup.com/pycon_11acef6aeb.png)

More than 200 people took part in the survey, and over a thousand responses were collected. The experiment process and its results were also presented at an academic conference. From an engineering perspective, this project showed that even a simple application reveals many operational concerns the moment you start considering real deployment: a clear interface, consistently structured data collection, quick deployment, and readiness for traffic spikes on the day of the event all mattered.

This experience also connected back to the broader theme of the internship. Whether the task is benchmarking GPU workloads or collecting human preference data, the system around the result matters. Tooling, deployment, data storage, and user experience all shape the quality of the final outcome.

## Open Source and Community Context

Lablup's engineering culture also gave me a closer look at how open-source communities and infrastructure products influence each other. AI infrastructure does not exist separately from the broader cloud-native ecosystem. It builds on shared ideas around containers, orchestration, scheduling, observability, and reproducible operations.

I had the opportunity to attend KubeCon + CloudNativeCon 2025, where Lablup was a sponsor. The conference made the connection between cloud-native infrastructure and AI infrastructure much clearer. That experience also led to my first open-source contribution to Kubernetes later on. I still actively contribute within the SIG Docs community, and I have been exploring cloud-native technologies more deeply by running a small homelab built with Raspberry Pi clusters.

Later, Lablup supported Emory University's CS hackathon by providing GPU resources and access to Backend.AI. In that environment, the importance of onboarding and usability became very clear. Participants had to listen to my short explanation, understand the platform within a limited amount of time, run workloads, and troubleshoot issues.

![Emory University hackathon](https://cdn.lablup.com/Hackathon_images_2a0b7c23eb.png)

These experiences helped me see infrastructure platforms as interfaces between complex computing resources and the people who want to use them. Good infrastructure should make powerful systems more accessible. I also came to understand how much Lablup values building a culture that actively encourages open-source contribution and community participation.

## Looking Back

My internship at Lablup changed how I think about AI engineering. Before, I mostly approached AI through the lens of model development. Working on benchmarking, FastTrack contributions, event page development, and open-source workflows showed me that the infrastructure around models is just as important as the models themselves.

The biggest takeaway was that AI infrastructure requires measurement, automation, usability, and community feedback together. Reliable systems are not built only by optimizing performance. They also require making results interpretable, tools usable, and workflows repeatable.

For anyone entering the AI field, it is easy to focus only on model architecture and training techniques. Those topics are important, but they are only part of the picture. As AI workloads become larger and more widely deployed, the systems that support them will continue to shape what teams can actually build, test, and operate.

As a research intern at Lablup, Taeeun gained experience with Backend.AI, FastTrack GPU benchmarking, open source contributions, and PyCon event page development. 

Internship Inside

Beyond model training, experiencing AI infrastructure

Yonggeun Kwon

## Overview

In May 2024, I came across a Facebook post from Lablup announcing their internship recruitment. I applied through the Typeform link provided. I still vividly remember the struggle of writing in English for the Typeform (even though it stated that either Korean or English was acceptable, I felt I have to write in English). Since Lablup develops **Backend.AI**, an open-source software platform, I thought technical development skills would be essential. However, as my studies had been focused on AI research, I confidently applied to the research team.

The interview was conducted in English with three interviewers: Sergey and Eunjin from the research team, and Kyujin from the DevOps team. I did my best to explain the projects in my portfolio in English, and before I knew it, 40 minutes had flown by. Although I felt I couldn’t fully convey my prepared points due to my limited English conversation skills, I unexpectedly received an acceptance email three days later.

That’s how I began my internship in July with three other interns. Lablup was my first experience in a corporate, so I tried to dress as neatly as possible for my first day. I wore a long-sleeved shirt and slacks despite the summer heat. When I arrived at the office just before 10 a.m., I was surprised to find Jong-eun, dressed casually in shorts and sneakers, as the only person there. He explained, "There are no assigned seats here, so feel free to sit wherever you’d like," and mentioned that others were working remotely. During the 10 a.m. all-hands meeting, I got to meet the team through Jong-eun’s screen. The atmosphere was quite different from the corporate image I had in mind, which initially caught me off guard. However, through self-introductions and coffee chats, I gradually adapted to Lablup's unique environment.


## Work
### Developing a Domain-Adaptive Language Model

After onboarding (installing Backend.AI), I joined the research team to begin my first task: developing a language model specialized for the trade domain.

 ![](https://cdn.lablup.com/Developing_a_Domain_Adaptive_Language_Model_image_6faf49194c.png) 

The general workflow is depicted in the diagram above. The goal was to train a language model to extract keywords and summaries from trade-related email conversations. These outputs would then help customers generate quotations. Detailed information about the model development and evaluation process can be found on the [Lablup blog](https://www.backend.ai/ko/blog/2024-09-26-finetuning-with-fasttrack).

To develop this language model, the process involved several steps: [Dataset Collection and Preprocessing] → [Model Training] → [Evaluation]. Throughout this process, I encountered many questions and faced initial challenges working in the Backend.AI environment. Since the internship was only two months long, every day felt precious. If I got stuck on something, it could cost me an entire day, so I quickly learned the importance of asking for help without hesitation.

One of the best thing of Lablup was its asynchronous communication style. Whenever I had a question, I could post it on the relevant channel. While it felt intimidating at first, knowing everyone could see my question, it was far better than wasting time being stuck. The team members were always kind in their responses, which helped me save time and work on other tasks while waiting for answers. This open-source culture of collaboration was truly a strength of Lablup.

### Evaluating the Agent Helmsman with LLM

As mentioned earlier, Lablup develops **Backend.AI**, a software platform with numerous CLI-based commands. For new users, memorizing all the commands can be challenging. While there is a WebUI, even this can feel unfamiliar for first-time users.

**Helmsman** is an LLM-based agent designed to address this issue. Users can provide natural language instructions (e.g., "Create a session" or "I want to train the Llama 7b model") through a chat interface. Helmsman then either provides the appropriate CLI command or executes it directly. This allows users to achieve their goals through simple chat instructions, making the platform much more accessible.

However, Helmsman didn’t always provide accurate responses. Since it relied on Backend.AI’s CLI documentation and commands, outdated or incorrect documentation could lead to errors. Additionally, as an LLM-based agent, it was susceptible to hallucinations. To evaluate Helmsman’s performance and identify issues in the documentation, I developed the following LLM-based evaluation system after discussions with Sergey and Eunjin.

 ![](https://cdn.lablup.com/LLM_image_c3b2c075a0.png) 


#### 1. **LLM1: Instruction Generator**

The first step is performed by the Instruction Generator LLM. This LLM analyzes Backend.AI's documentation to generate user instructions that are likely to be used in Backend.AI.

For example, it might generate an instruction like, "Create a session with 4 CPU cores and 16GB of memory."

---

#### 2. **LLM2: CLI Command Generator**

The second step is performed by the CLI Command Generator LLM. This LLM generates CLI commands based on the instruction created by the first LLM, along with the CLI documentation and a few-shot examples.

For instance, it might generate the following command:

```
'[Backend.AI](http://backend.ai/) session create
--name cli-test-session --resources cpu=4
--resources mem=16g --resources cuda.shares=2 ,…’
```

---

#### 3. **Executing and Logging CLI Commands**

The generated CLI command is executed, and the results are logged as follows:

- **Success/Failure Status:** Checks whether the command executed as intended.
- **Error Logs:** Records error messages if the command fails.
- **Reference Documentation:** Logs the documentation used to generate the command.

Through this process, users can verify the results of command execution and, if issues arise, obtain additional information to troubleshoot the problem.

### Result Analysis

 ![](https://cdn.lablup.com/image_f83afde9c3.png) 

Errors can be categorized into three types:

1. **Instruction Errors:** Caused by insufficient user input, which can often be resolved through user-agent interaction.
2. **Documentation Errors:** Rare but critical, as they result in repeated failures. These require corrections in the documentation.
3. **Hallucination Errors:** Mitigated by adding few-shot examples. However, prompt length becomes a concern, so matching few-shots to specific commands can help.

Due to time constraints, the tasks following the result analysis were left as future work.

## Reflection

Started as a two-month internship was extended to six months. Looking back, I realize that two months might have been too short to fully experience and contribute to Lablup, given the depth of Backend.AI and the remote-friendly culture.

However, as mentioned earlier, Lablup encourages a culture of asking questions. By taking advantage of this, even two months can be enough for meaningful growth.

 ![](https://cdn.lablup.com/_5f63ce07ce.png) 
 ![](https://cdn.lablup.com/_34f96619ed.png) 

Beyond work, I also had the chance to participate in events like PyCon and the Lablup Conference. Notably, I even had the opportunity to be a speaker at the Lablup Conference. Though I felt there was much to improve in my presentation, it was an invaluable experience.

Lablup is a company filled with unique individuals. Everyone’s character stood out so much that I often thought of myself as the most ordinary person there. Yet, I found that their distinctiveness also came with extraordinary talents. I laughed a lot, learned a lot, and received a lot of help. This made work enjoyable, and I never dreaded going to the office.

My time at Lablup was not only a period of professional growth but also a time of personal enrichment through meaningful interactions and experiences. I hope to carry forward the values I learned here and continue to grow.

A six-month 2024 internship at Lablup, building domain-adaptive language models and evaluating an LLM agent for CLI automation, with talks at PyCon along the way.

Recap of my Lablup Internship, 2nd half 2024

Byeongjo Kim

# Overview

I applied to the Open Source Contribution Academy (hereinafter referred to as the Contribution Academy) hosted by OpenUp, and worked as a fall intern at Lablup Inc. (hereinafter referred to as Lablup) from November to December for 8 weeks. Afterwards, I extended it for an additional 8 weeks from January to February, working a total of 16 weeks.

After being discharged from the military, I have written about the experiences I had while working at Lablup, my first company as a developer.

# Motivation for Applying

Even before the Contribution Academy, I was interested in Lablup, and coincidentally, I had an opportunity to contribute through the Contribution Academy.

During the Contribution Academy period, I worked on resolving issues and refactoring the webui of Backend.AI.

While participating in the Contribution Academy, I felt a lot of affection, interest, and enjoyment towards Backend.AI, and I began to think that I wanted to continue contributing after the program ended.

Lablup happened to provide an opportunity to work there in conjunction with the Contribution Academy, so I applied without hesitation.

# Onboarding

For the first 3 weeks of the internship, I underwent an onboarding process.

I went through implementing a RealTime Chat, setting up the Backend.AI environment, and then the Pebble Seminar in that order.

 ## RealTime Chat

This was the first assignment to become familiar with the core side of Backend.AI's code. I implemented a real-time chat app using Python, utilizing the aiohttp, aioredis, and asyncio libraries.

Since there was no condition to store the chat contents, I used the in-memory database redis.

I made it so that when a user enters a chat room, they subscribe to that chat room, and when a user enters a message, it publishes the entered message to the subscribed users, meaning the other users in the same chat room.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_01_f466f06469.png"/>

 RealTime Chat in action


While I was able to handle Python at a basic level from preparing for coding tests, I had no experience using libraries like aiohttp, asyncio, and aioredis, so it took me some time to understand and grasp the concepts.

However, this assignment helped me a lot in understanding the core side of Backend.AI's code, and it was good to be able to study new libraries.

## Setting up the Backend.AI environment

Since I had already installed Backend.AI during the Contribution Academy, setting up the environment during the internship period wasn't too difficult.

However, I was well aware that installing Backend.AI is not easy, as I had encountered many errors and failures while trying to install it during the Contribution Academy, and the other person doing the internship with me also had a lot of difficulties during the installation process.

Since I had already experienced those failures and knew the solutions, I was able to help, and we were able to install it quickly and move on to other tasks.

While setting up the environment, I also configured a virtual machine and VPN, and set up the environment on a virtual machine as well, so that I could work even if there were problems on my local machine. After setting up the configuration on the virtual machine, I mainly used the local for development during the subsequent work, and the virtual machine as a test server. The company's VM Farm, which allows for easy management and configuration of virtual machines, made it great for setting up development and testing environments.

## Pebble Seminar

After completing the RealTime Chat and setting up the Backend.AI environment, I prepared a short seminar based on understanding the structure and code of Backend.AI. I was tasked with presenting on GraphQL and Relay, which are used in the Backend.AI WebUI.

While I had experience with GraphQL, I felt that my knowledge was lacking for presenting in front of others, and Relay was a new library to me, so I was quite worried about preparing for the Pebble Seminar and read through many documents to prepare. First, I familiarized myself with the concepts by reading the official documentation for GraphQL and Relay, and then analyzed the Backend.AI code one by one to understand how they were applied and functioning in Backend.AI.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_02_25c7f67c06.png"/>

 Pebble Seminar preparation materials


By analyzing the code while preparing for the Pebble Seminar, I naturally came to understand the code running in the WebUI, and this greatly helped me in finding and resolving issues during the subsequent work.

# Resolving Backend.AI issues and implementing features

After completing the onboarding, I finally joined the frontend team and started resolving Backend.AI issues and implementing features. I had a coffee chat with the frontend lead to define the categories of work for this internship period:

1. Creating a Table Column Setting component
2. Researching E2E Testing
3. Daily tasks

During the 8-week internship period from November to December, I created a total of 19 Pull Requests, of which 18 were merged, and 1 is still under review. Since I had experience finding and assigning issues during the Contribution Academy, I had less difficulty with it, and because I enjoyed resolving issues, I was able to resolve more issues than others.

## Feature Addition PRs
### 1. Implementing Table Columns Setting

https://github.com/lablup/backend.ai-webui/pull/2071

This was one of the issues I aimed to work on during the internship period. It was the only component that I conceived and implemented from scratch during the fall internship, rather than refactoring an existing component. Before implementing this feature, I thought it was a simple task that I could finish quickly, but things turned out differently from my expectations.

First, I realized that I had been thinking too simplistically about creating new components. Even though I had designed and considered the props to be received before creating components in the past, through this issue, I felt that when creating a new component, I should invest more time and effort while considering scalability. I also realized that I should pay more attention to how other sites are designed and what features are applied.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_03_b4d7b20042.png"/>

 Table Columns Setting


### 2. Adding service endpoint and owner columns to the Model Serving page table

https://github.com/lablup/backend.ai-webui/pull/2047

Previously, when creating a model service, users had to go to the detail page to check the endpoint, which is a frequently used feature. So there was a request to add the endpoint to the table column. Additionally, since the admin account can see services of users in the same group, there was a suggestion to have a column showing the service owner. Since the GraphQL fields for retrieving this data were already implemented, I added the fields to the query to fetch the endpoint and service owner data, and then added columns to the table to display the data. The owner column is only shown for admin accounts.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_04_cd838ef9b2.png"/>

 Implementation view. Screen for admin account (left) and user account (right)


### 3. Disabling log button for sessions in CANCELLED state

https://github.com/lablup/backend.ai-webui/pull/2045

The CANCELLED state means that the container has never been created or failed to be created. Previously, the log button was enabled even for sessions in the CANCELLED state, and if a user clicked the log button, the agent could not find the container information, resulting in a 500 error. In this PR, I made it so that the log button is disabled for sessions in the CANCELLED state, preventing users from clicking it.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_05_973936500c.png"/>

 Session in TERMINATED state (session 1) and CANCELLED state (session 2)


### 4. Testing and creating a custom hook for dark mode

https://github.com/lablup/backend.ai-webui/pull/2120

Before implementing dark mode, I found components with hardcoded colors and implemented a custom hook named useThemeMode for applying dark mode. When creating the custom hook, I tried to use the useLocalStorageState hook from ahooks, but contrary to my expectations that it would automatically manage states with the same key value, I found that they operated independently. To handle states with the same key value automatically updating when the value changes, I added a custom hook named useLocalStorageGlobalState, and then used that to create the useThemeMode custom hook for setting the dark mode.

## Bug fix PR

### 1. Allowing signup without invitation token

https://github.com/lablup/backend.ai-webui/pull/2046

In the config.toml, when the allowSignupWithoutConfirmation option is set to true, users can sign up without an invitation token. However, when a user clicked the sign up button, an error occurred because the token value was undefined. In this PR, I modified it so that if allowSignupWithoutConfirmation is true, the token variable is not used. Additionally, previously users could modify other input values while the core was processing the data after clicking the sign up button, and the previous data remained when the dialog was closed and reopened. In this PR, I made it so that other input values cannot be entered while data is being processed, and the previous input values are cleared when the dialog is closed.

### 2. Displaying the correct screen for the selected sub-tab on the user management page

https://github.com/lablup/backend.ai-webui/pull/2055

On the user management page, there are sub-tabs for displaying active users and deactivated users. However, if a user navigated to another page and then returned to the user management page, even though the sub-tab was set to inactive, the screen displayed the list of active users, causing confusion. This PR resolved that issue by remembering the current sub-tab when navigating to another page, and displaying the appropriate screen for that sub-tab when returning to the user management page.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_06_c206b10b14.png"/>

 Before (left) and after (right) the fix


# Extending the internship

As I resolved issues, the 8-week period flew by, and it was time to wrap up the fall internship.

Working at Lablup as my first job after being discharged from the military was an important period for me. During the internship at Lablup, I was able to experience my strengths and weaknesses, what skills I needed to further prepare, and how other developers work. The 2-month period felt very short, and since I had enjoyed working so much during that time, I wanted to continue working. So I expressed my desire to extend the internship to the lead, and we agreed to extend it for another 8 weeks until February. During the fall internship, I had thought a lot about my weaknesses, but I couldn't find my strengths. So, I started with these 3 personal goals:

1. Find my strengths during this period
2. Read the documentation whenever I have time
3. Work even harder, leaving no regrets

# Resolving issues and implementing features during the extended period

The work during the extended period did not differ much from before. Without the onboarding process and installation, I could focus more on resolving issues.

## Feature Addition PRs
### 1. Refactoring ErrorLogList

https://github.com/lablup/backend.ai-webui/pull/2131

I refactored the ErrorLog List, which was previously implemented using Lit elements, to React. This feature was the most satisfying issue for me, as I personally use it frequently after the refactoring.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_07_11ad8297d0.png"/>

 Before (left) and after (right) refactoring


During the refactoring, new Search and Error filter features were added.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_08_fb17ed6a93.png"/>

 Added Search feature (left) and Filter feature (right)


### 2. Modal drag functionality

https://github.com/lablup/backend.ai-webui/pull/2179

I used the React-draggable library to add functionality for modals to be dragged. By adding the Draggable prop to a modal, it can be applied to any modal that requires dragging.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_09_e0ee20954a.png"/>

 Draggable Modal


By clicking the icon on the left side of the modal title and moving the mouse, the modal can be moved to the desired position on the screen.

Currently, it is applied to the modal for viewing user information on the user management page and the modal for changing user settings, where you can check it.

<iframe title="'일하는 개미의 개발..'에서 업로드한 동영상" width="640" height="360" src="https://play-tv.kakao.com/embed/player/cliplink/444997096?service=player_share" allowfullscreen frameborder="0" scrolling="no" allow="autoplay; fullscreen; encrypted-media"></iframe>

While it is not being used in many places yet, I think this PR will be useful as more components and features are added.

## Bug fix PR
### 1. Modifying Vfolder invitation permissions

https://github.com/lablup/backend.ai-webui/pull/2143

There was an issue where the user permissions for group vfolders were not being updated. When trying to modify the permissions, the items were not displayed or selectable properly in the select. Previously, the items were being displayed using option tags, but I changed it to use mwc-list-item to display the items and modified the overflow option to resolve this issue.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_10_bf182d8b57.png"/>

Before (left) and after (right) the PR


### 2. ResourceGroupSelect extending outside the card

https://github.com/lablup/backend.ai-webui/pull/2166

There was an issue where the ResourceGroupSelect value would be displayed outside the card if it was too large.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_11_e1038e9aec.png"/>

Symptoms of the issue


To resolve this issue, I set the max-width CSS on the Select component so that it cannot exceed the width of the card.

Additionally, in this PR, I added a Search feature to the Select component, for which I used the useControllableValue hook from ahooks. The useControllableValue hook helps manage props from either the parent or itself. While it was a simple PR, it took me longer than expected since it was my first time using useControllableValue. I was able to resolve this issue with the help of the lead and another intern.

### 3. Key pair list not showing when clicking the generate & manage key pair button on the summary page

https://github.com/lablup/backend.ai-webui/pull/2194

On the summary page, there are buttons for "Generate New Key Pair" and "Manage Key Pairs." However, when clicking these buttons, instead of showing the key pair list, it simply navigated to the user management page, displaying the user list.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_12_ace27a911e.png"/>

"Generate New Key Pair" and "Manage Key Pairs" buttons on the summary page


<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_13_5e99f8c028.png"/>

When clicking the "Generate New Key Pair" button (left) and when clicking "Manage Key Pairs" (right)


While this issue was not critical, I resolved it because I had experienced a lot of confusion when I first used Backend.AI and didn't fully understand the key pair feature.

After resolving this issue, I could confirm that the key pair list was displayed on the screen as intended.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_14_429c2582f9.png"/>

After resolving the issue, when clicking the "Generate New Key Pair" button (left) and when clicking "Manage Key Pairs" (right)


# Completing the Internship

Thanks to the Contribution Academy, which started from a friend's recommendation after being discharged from the military, I was able to contribute at Lablup for an extended period. Since I had no previous internship or project experience at other companies, this was a very important period for me as I was starting anew after being discharged. It was great that I could experience my strengths and weaknesses, the skills I lack, and the culture of an open-source company at Lablup. How many companies make you want to go to work every day with their horizontal structure, free atmosphere, pleasant working environment, and good equipment? Although I worked at Lablup for 4 months, I felt like I wanted to go to work every day, and if it was Lablup, I could work at a company while doing interesting and desirable work for a long time. Over the 4-month period, I also developed a fondness for Backend.AI, the service provided by Lablup, and I plan to attend the conference hosted by Lablup every year whenever possible to see its advancements and technologies.

<img style="margin: auto;" src="https://cdn.lablup.com/intern_kbj_15_45032e1593.jpeg"/>

Lablup Office


> This post was also published on the author's personal blog.
> https://gee05053.tistory.com/32
> This post is automatically translated from **Korean**

A 2023 winter internship at Lablup: 16 weeks that began with the Open Source Contribution Academy, spent resolving and refactoring Backend.AI WebUI issues.

Recap of my Lablup Internship, Winter 2023

Dongjin Park

#Overview

I applied to CUop, a collaboration between universities specializing in science and technology, and worked as an intern at Lablup for 8 weeks.

I wrote about my experiences through onboarding, developing Backend.AI, and attending PyCon.

# Motivation for applying

I first learned about Lablup through a PyCon presentation session I stumbled upon. I could tell that the company had a lot of technically deep and passionate members. I applied to Lablup because I was interested in Python and asynchronous programming.

# onboarding

During the first two weeks, we conducted an onboarding process.

We went through the Realtime Web Chat implementation, Backend.AI development environment, and code base seminar.

## Realtime Web Chat

This is an assignment to familiarize yourself with Python asyncio. You will develop a real-time chat app using aiohttp, an asynchronous web framework, and redis, an in-memory database. The task also includes setting up docker compose to build python and redis at once. For more information, see the [Github Readme](https://github.com/pderer/webchat).

To broadcast the messages through redis, we used Redis pub/sub. pub/sub acts as a platform that delivers messages without storing them. Since we didn't have a requirement to store messages, we used Redis pub/sub. We also registered the Redis pub/sub process as a task using asyncio.create_task() to run it in an event loop.
![Realtime Web Chat Launch Screen](https://cdn.lablup.com/Realtime_webchat_0f87103fe6.png) 
Realtime Web Chat Launch Screen

I was able to understand the basic behavior of asyncio. I was able to ask questions and solve the difficult parts. I think Lablup has a great environment for interns and junior developers to grow because they can freely ask questions through Microsoft Teams.

## Build the Backend.AI development environment

I installed Backend.AI on a VM Farm and a local VM and tried to run it myself. I read the official documentation, but the process was not smooth. I encountered various errors, which I solved by sharing with my fellow interns and asking questions on Teams.

> 💡 A VM farm is an environment where virtual machines are managed and run. Lablup has its own VM Farm.

This was my first experience of developing with a VM Farm and VSCode connected via SSH. To develop Backend.AI, I need to run multiple processes (Manager, Agent, Storage proxy, Web server, etc.) and Docker Containers. If I use a laptop, it's a bit of a battery drain. However, with VM Farm, you can develop Backend.AI lightly because all you need is an SSH connection. In fact, I was able to develop for a long time using VM Farm when I was out of the office and couldn't charge my laptop.

## Code Base Seminar

After looking at the code, focusing on the difficult parts of Backend.AI, I prepared a seminar based on my understanding. I was in charge of presenting the Manager, Agent, and GraphQL parts.

Since Backend.AI is open source, the official documentation is well written. I studied the architecture of Backend.AI by looking at the official documentation to understand the overall structure and asking the employees directly if I had any questions. Since the topic of the presentation was session/kernel creation and scheduling control of Backend.AI Manager, I studied the manager code and analyzed the logs of the manager process.
 ![A sequence diagram I drew in preparation for a seminar presentation.](https://cdn.lablup.com/Sequence_Diagram_27d4a0128e.png) 
A sequence diagram I drew in preparation for a seminar presentation.

Analyzing the logs, I found a bug that caused the session state to change from Preparing back to Pulling. It was rewarding to analyze the logs one by one. It was difficult to analyze the logs in order because of the asynchronous code base, but drawing a call graph and a sequence diagram was very helpful.

# Develop Backend.AI

After onboarding, I started working on Backend.AI. I worked on GitHub issues and volunteered to help or found and resolved issues myself.


I created 9 pull requests from the [Backend.AI repository](https://github.com/lablup/backend.ai/pulls?q=is%3Apr+is%3Aclosed+author%3Apderer) repository and 2 pull requests from the [Backend.AI-WebUI repository](https://github.com/lablup/backend.ai-webui/pulls?q=is%3Apr+is%3Aclosed+author%3Apderer), and they all merged!

I chose high-priority issues that I was confident in addressing. I wanted to make as many contributions as I could in the short time frame of two months.

## First PR

https://github.com/lablup/backend.ai/pull/1395

I created a PR to fix a bug I found while preparing for a seminar. It was an easy PR to fix the API parameter. However, it was a good experience to learn about branch name convention, commit convention, and news fragment, and to experience the CI (Continuous Integration) process with GitHub Actions and to do some Git-related shoveling beforehand.

> 💡 A news fragment is a one-sentence Markdown description of what the branch created by the PR is trying to do. You want to keep it simple and clear so that if you see this PR again in the future, you'll know what it was trying to do.

## PRs for vfolder

When I heard that Teams had an open issue for an intern, I jumped at the chance to apply. I had to learn a new concept called vfolder, but I knew it would be important to get to know the product.

### PR (1)

https://github.com/lablup/backend.ai/pull/1397

Only admin can create a project type vfolder. It should be possible to create a vfolder regardless of the max_vfolder_count of the keypair resource policy, but if the user type vfolder exceeds the max_vfolder_count, the project type vfolder cannot be created. At first, I was confused by the terminology, but by analyzing the code and asking questions, I was able to interpret the terminology.

### PR (2)

https://github.com/lablup/backend.ai/pull/1400

Fixed new bugs discovered while addressing PR (1).

### PR (3)

https://github.com/lablup/backend.ai/pull/1417

I found an issue related to the PR (1) issue. DB migration and GraphQL were new to me, but I wanted to try them out, so I supported them. I used a DB migration tool called [Alembic](https://alembic.sqlalchemy.org/en/latest/), studied GraphQL schema concepts, and modified my query and mutation code to support backward compatibility. I tried to use cURL to test the modified code, but GraphQL has a much longer request format than the REST API, which was cumbersome. I wrote the test code by asking questions to interns and employees who are familiar with GraphQL. I wrote python code to test the modified query and mutation in the form of CLI to make testing easier.

## WSProxy related PRs

Supported an issue in Teams. There was a bug in the WebUI where it was not possible to delete a session if the wsproxy address of the resource group was invalid. I also wanted to get some experience with WebUI development.

### PR (1)

https://github.com/lablup/backend.ai/pull/1423

I read through the WebUI code to troubleshoot the issue, but I couldn't quite grasp the concept of wsproxy. I realized that wsproxy has v1 and v2, but it was not easy to understand the difference between the two, so I asked the employees. The main difference between v1 and v2 is the path of the traffic: v1 goes through the manager to communicate with the container, while v2 can communicate directly with the container without going through the manager, which is faster. Once I understood what wsproxy does and the difference between v1 and v2, it was easier to understand how the code worked, and I realized that a lot of people didn't know the difference. I realized that questions that seemed easy to ask might not have been asked in the organization.

### PR (2)

https://github.com/lablup/backend.ai-webui/pull/1819

I also modified the webui code to fix the issue. To fix the JavaScript code, I learned about callback functions, promise objects, and async/await. I handled errors so that they didn't affect other logic, and defined duplicate code as functions to eliminate duplication of code.

### PR (3)

https://github.com/lablup/backend.ai-webui/pull/1833

However, since the WebUI needs to be backwards compatible with Backend.AI 22.09, we had a review from our CEO that it should also handle HTTP Status 404, so we made it handle both 404 and 500.

### PR (4)

https://github.com/lablup/backend.ai/pull/1466

However, after the code was merged, a bug occurred: when setting up v1 wsproxy, the return value for wsproxy-version disappeared. This happened because I was modifying the core code and didn't handle all the branches. I fixed the code in a hurry, but it was a simple mistake, and I realized that I should write test code to prevent such mistakes.

## PRs related to Manager

https://github.com/lablup/backend.ai/pull/1444

In preparation for the seminar, I came across an issue with manager that I had been studying. With my internship coming to an end, I thought I could contribute by resolving the issue on the code I knew best.

This PR has been heavily modified by code reviews. Initially, we designed scheduler health check and scheduler trigger as the same API. After receiving code reviews, I split the two functions into different APIs to separate responsibilities. We originally stored health information only for the schedule function, but we also stored health information for the prepare function and the scale_services function because we needed to create the trigger API after storing health information for the three functions that run periodically according to the scheduler's global timer to get a complete picture of the scheduler's health. We also changed the design so that we can store the scheduler's health based on the manager ID because there may be multiple manager processes.

The scheduler's storage was also reviewed. Initially, we looked at the code in the existing manager state API to store manager state in etcd and did the same for scheduler state. etcd is good for storing configuration information that needs to be kept consistent, but it is slow to write. redis is a volatile database, but it performs well even with many reads/writes. Since the scheduler state is read/written periodically and does not need to be consistent, we switched to storing it in redis.

## Agent-related PRs

https://github.com/lablup/backend.ai/pull/1472

Now that I had a good understanding of the Manager part of Backend.AI, I wanted to understand another important component: the Agent. I came across an issue about the Agent, so I took a look.

While Backend.AI was running, there was a bug where the internal state of the Agent did not match the state of the actual working container. As a result, when creating a session, an InsufficientResource Error was thrown during the resource allocation process, even though there were actually enough resources. When the error occurred, we needed to improve the logging to understand what went wrong during the resource allocation process.

It took a while to figure out the resource allocation process. The concurrency issues were difficult, and it took a lot of Q&A with the CTO to get a general idea of the flow and what to log.

A few weeks after my internship ended, the CTO merged it with more than 10 commits, adding refactorings and test code. What impressed me was that he wrote test code to reproduce the error. I had to go through a complex process (see PR) to reproduce the error myself, which took a lot of development time. I could see the difference in productivity here. Of course, I thought about writing test code, but I thought the implementation would be too complicated, and I thought that writing test code would be the end of my internship. In the future, I shouldn't be so intimidated by writing test code and just try it out and learn as I go. Also, it seems that the refactorings were done with a focus on code readability. The functions in the part I modified were too long and not very readable, but after refactoring, the functions were shorter and the logging was cleaner. I realized that I shouldn't stop at implementation, but try to make good code.

# Attending PyCon

On August 12 and 13, Lablup will have a booth at PyCon. Companies that sponsor PyCon are given the opportunity to have a booth. I was an intern, but I wanted to participate in the booth activities and listen to some of the talks. The company had some PyCon tickets left over, so I was able to participate.

At the Lablup booth, we had an event where we challenged Llama2 to write a 10-line pyramid at a prompt. The challenge wasn't that easy, and it was important to explain it in a way that Llama2 could understand. Two lucky people who submitted the correct answer were drawn to win a Nintendo Switch and a Logitech mouse. My role at the booth was to help direct PyCon attendees to the event, and since there were a lot of employees at PyCon, I was free to attend any talks I wanted to hear. Since Rableup is an open source company, we encourage people to contribute to open source and participate in conferences. In fact, four of our presenters attended PyCon, so we value participating in conferences.

 ![Lablup Inc. booth](https://cdn.lablup.com/pycon23_booth_6059bf7e37.jpeg) 
Lablup Inc. booth

During the RustPython session, a tool called ruff was introduced as a replacement for the python lint tools flake8 and isort. Since ruff is composed of Rust, it is 100x faster than flake8. At Backend.AI, we were using flake8 and isort for lint, but after our CTO reviewed ruff, I watched him adopt ruff for the Backend.AI project right on the stairs of COEX. I realized that he was really good at the process involved in coding, applying new tools to the project in a short time and even revising the official documentation. I realized that I want to be a proficient developer someday. After PyCon, I read the updated official documentation, applied ruff to the Backend.AI development environment, and experienced linting 100x faster. If I hadn't participated in PyCon, I wouldn't have gotten up to speed on these great tools so quickly. I hope to continue participating in developer conferences in the future.
![Group photo with members of Lablup Inc.](https://cdn.lablup.com/pycon23_all_dee19bbc26.jpeg) 
Group photo with members of Lablup Inc.

# End of internship

During my internship, I tried to get as much experience as possible, and I wanted to contribute a lot. In the end, I was able to experience a lot because I tried to contribute a lot. I was quick to volunteer for issues that came up in Teams, so I was able to understand the core components of Backend.AI: vfolder, wsproxy, web-ui, manager, and agent. I also learned new concepts like DB Migration, GraphQL, and etcd. Although it was a bit physically demanding to attend the conference from morning to evening on the weekend, it was fun to listen to more than 10 presentation sessions, get inspired, and meet various people through booth activities.

During my internship, I think I actively asked questions about things I didn't understand, which helped me to solve issues quickly. I think the reason why I was able to ask a lot of questions was because there was a culture of horizontal rabble-rousing, and there were many people who were kind enough to answer my questions, so I was able to actively ask questions. I would like to take this opportunity to thank the members for their support.

I was able to experience a variety of things, including asynchronous programming experience, GitHub collaboration, presenting English seminars, and attending conferences. I feel like I've grown a lot as a developer through this program. I recommend the Lablup internship to anyone who is thirsty for growth.

> This post is automatically translated from Korean

A 2023 summer internship at Lablup through the CUop program: eight weeks of asyncio onboarding projects, Backend.AI development, and a trip to PyCon.

Recap of my Lablup Internship, Summer 2023

SeongHyun Koo

I first encountered and started working with Backend.AI through the 2022 Open Source Contribution Academy (hereafter OSSCA). After the program, I wanted to get more seriously involved in the project, so I applied for the OSSCA corporate-linked internship and had the opportunity to intern at Lablup for about two months. In that process, it wasn't easy for a beginner who had just set foot in such a large project to contribute code. Nevertheless, I want to talk about the contribution experiences I was able to accomplish little by little and what I gained from them.


## Technical Challenges

**Onboarding**

The first hurdle in contributing to the project was onboarding. Since I had participated in the mentoring program with the Backend.AI team during OSSCA, I thought I could start the internship with a decent understanding of the structure. However, there were so many new things to learn that what I had seen so far felt like just a tiny fraction of the whole. I understood the role of each component, but the variety of technologies used within them made the code difficult to read. Around the time I was feeling this way, I was given an assignment as part of the onboarding process: to write an asynchronous web chat program. So, for about a week, I worked on the assignment using several tools like Python asyncio, aiohttp, and redis. In the process of trying to use the tools employed in the project, I naturally started to gain an understanding of them. Based on what I learned from the assignment, I was able to read the base code and understand its role.

<figure>
 <img src="https://cdn.lablup.com/1_090323f7f3ea330cda4718b950188559_e3464cc987.jpg" alt="Development environment setting">
 <figcaption>Development environment setting</figcaption>
</figure>

### Setting up the development environment

The second hurdle was setting up the environment. I had experienced this during OSSCA, but I inevitably faced it again during the internship. To contribute code, you always have to work with the latest version, not a stable one, so even if you set up the environment properly once, you might face new errors during an update. However, there was an area where I had improved. Before the internship, if an error occurred, it was difficult to figure out when and where it happened. But now, after learning about the libraries, tools, and technologies being used, I could identify which component's function was failing or where a conflict was occurring. To use an analogy, if before I didn't know what to do to untangle a knotted skein of yarn, now I felt like I had learned how to find the beginning of the tangled mess and unravel it bit by bit.


 ![](https://cdn.lablup.com/Untitled_73d5277b5637793ce6f5504fa2df9efa_aa81b1fea1.png) 

### Tasks
#### Good first issue

After the onboarding period, I was assigned about three issues that are often called "good first issues," which are simple and don't require extensive work. One of them was to implement the 'feature to selectively load plugins,' which was already implemented in the agent, on the manager side as well. So, I started to break it down step by step. I looked at what the TOML-formatted configuration file could pass, at what point it was loaded into memory, and where type checking was done. I also referred to issues and code written by others to understand the logic for how plugins are loaded and blocked, and then I gradually wrote the code, contemplating how to apply it to the manager side. Since Backend.AI is a real product in use, I was very cautious at first, even when writing a single line. I was very concerned about whether there might be better code than what I wrote, or if it would be difficult for others to read due to poor readability. However, as I worked on this issue, I inserted my own code into hundreds of lines of existing code, and I revised and improved the code based on the code review. Going through the entire process up to the merge made it feel much more natural.


 ![](https://cdn.lablup.com/12_56_48_52b90d30e660087bca2ced52717fd10c_4eee1be16b.png) 

#### Monitoring

After finishing the "good first issues," I was looking for what I could improve next. I saw an issue about improving the monitoring-related infrastructure and found it interesting, so I chose it as my topic. Before my internship, when I imagined working in the field, I used to picture someone whipping up amazing and complex code in a flash. But in reality, a significant amount of time is spent reading existing code to understand its context or researching and studying the technical aspects used in that part. I also spent a lot of time researching the monitoring infrastructure and reading the existing code for my newly chosen issue. After that process, to monitor the resource usage of processes, I wrote agent backend code that gets a list of processes running inside a container using cgroup or the Docker API and collects metrics for resource usage like CPU and memory using third-party modules. The amount of code and the logic to be written became larger than expected, so I repeatedly contemplated how to integrate it well with the existing infrastructure, and later I made revisions based on the reviews I received. In that process, my interest in program monitoring deepened, so I later created and sent a PR for code that monitors the resource usage of Kubernetes nodes in the agent, a topic that wasn't part of an existing issue.

 ![](https://cdn.lablup.com/02_bf763fba7fd3799208ef693c6caf941d_215a4c584b.jpg) 

### Conclusion

The simple yet profound insight I gained from my internship and direct work experience was: "In the end, good code is not written by struggling alone!" No matter how much you think or research, the code written by one person ultimately stays within that person's perspective and field of vision. Even a program written by a true genius is bound to have errors. The process of communicating with other colleagues, sharing opinions, and filling in and improving what was lacking is necessary. I believe this is why code with good readability is emphasized. This reinforces the idea that you should write easy-to-read code so that both you and others can continuously develop the program, and that even a raw idea or code that might be a bit embarrassing is improved when it's the result of seeking help from colleagues and creating it together.

For me, as someone who was experiencing their first job, Lablup was a surprisingly horizontal and communicative place. Through daily all-hands meetings, everyone could learn about the company's business progress. When a problem arose, people helped each other and suggested new ideas to improve the situation via messenger, regardless of their position. I think that being able to participate in a project and work together in such a place for two months was a very precious and valuable experience.

A 2022 fall internship at Lablup, and the lesson that good code comes from working together, told through hands-on projects and team collaboration.

Recap of my Lablup Internship, Fall 2022

Sion Kang

## Introduction
My first encounter with 'Lablup' was in the summer of 2019. At that time, I attended a GDG Seoul 'Everyone's Toy Story' event because someone I knew was presenting there. I had the opportunity to sit in on a presentation by Lablup about their GPU virtualization tools for machine learning, and that piqued my interest. At that time, my fascination with machine learning was at its peak, and the technical depth of their presentation was impressive—it was my introduction to a company pioneering in this field.

 ![](https://cdn.lablup.com/202424596_37015a5e_8c90_4de4_af98_d9bb60047361_1389ca8bcb.png) 

Subsequently, I reconnected with the company at the 42 Seoul open source hackathon. This was an event focused on developing a product using a designated open source within a limited timeframe, where I became a part of the Backend.AI team. Despite it being three years since the initial presentation at GDG Seoul, the impression it left was so profound that the company's name was instantly familiar to me. Throughout the contest, the guidance provided by Jungkyu (CEO of Lablup) was instrumental, contributing significantly to our achievement being a second place at a hackathon.

In May 2022, I was working on a school-based internship at a company called SATREC INITIATIVE. As my internship was drawing to a close, I came across an announcement for Lablup's summer internship on Facebook. Recalling the valuable mentorship from Mr. Shin during a competition, and with a keen interest in the developer community and open source, I chose Lablup as my next destination.

At that period, I was engaged in a project named 42 World, which was a significant phase of learning and growth for me. In my interview with Lablup, I detailed my experiences with the 42 World project, particularly the challenges I faced while implementing a MonoRepo. Interestingly, Lablup had encountered similar issues with MonoRepo in Backend.AI, allowing us to exchange a sense of empathy as developers during our conversation.

After being accepted into the Lablup internship, I started my internship with four other interns. I joined the company about a week later than the others to give myself time to finish my existing internship and relocate. My first week was dedicated to orientation, familiarizing myself with Backend.AI, and acclimating to the company's culture. The onboarding documentation was comprehensive, contributing to a welcoming environment for newcomers to quickly adapt. A significant portion of my orientation involved installing and configuring Backend.AI. Since I began a week after the other interns, they were able to assist me, which made the orientation process relatively smooth.

## Getting to my work
During the second week, I began tackling the actual tasks. I opted to collaborate with the DevOps, Frontend, and Research teams. Each chapter leader provided me with a 'good-first-issue' to address. My choice was the DevOps team. My initial task involved refactoring the 'run' command, which initiates a session and runs the specified code. The goal was to integrate 'start', to launch the session, and 'exec', to execute the code, thereby minimizing redundant code.

 ![](https://cdn.lablup.com/202433613_91dd0878_8d6d_480a_b0ec_e3f637d7169b_6856417e93.png) 

I faced challenges with the first issue assigned to me as it required a thorough understanding of Backend.AI's repository structure, irrespective of the implementation difficulty. I realized that while it's important to know why an issue exists, it's also important to know exactly what Backend.AI is trying to accomplish and understand how the issue needs to be solved to achieve that goal so that I can solve it correctly.

After addressing the initial issue, I took on the task of testing a feature in development known as vfolder clone. My role was solely focused on DevOps, providing me with my inaugural experience with Backend.ai-webui, a Frontend chapter project. My involvement wasn't limited to just testing the vfolder clone; I also executed it personally, identifying areas for improvement and bugs, which I then documented as issues. This made me feel somewhat guilty, as it seemed I was generating work for other teams. However, the Frontend chapter's encouragement to contribute assuaged my concerns. This experience reaffirmed my understanding that open-source contributions extend beyond code; there are numerous ways to contribute.

 ![](https://cdn.lablup.com/202441600_3da0d632_abda_4953_ab42_c8c5379dbab2_82a74cd001.png) 

## Improving CI/CD

As I've always been interested in CI/CD, I was intrigued by the actions utilized in Backend.AI. At the time, Backend.AI could bypass CI with the `skip:ci` tag, but I noticed that neither `skip:ci` nor `skip:changelog` tags would work if the PR was labeled after creation, necessitating an extra commit. Since external contributors lack label permissions and Backend.AI is open source, resolving this seemed crucial. After exploring GitHub Actions, I discovered a trigger for labeling, which allowed me to address the issue. My proactive approach in finding and fixing problems independently, rather than through assigned tasks, was highly appreciated by the company. This experience deepened my interest in actions, leading to further enhancements. Noticing frequent omissions in assignments, I proposed the auto-auth-assign action for automation, which I had previously utilized. Subsequently, I considered automating the labeling process. Although new to the labeler action, I conducted multiple tests in a test repository before implementing it in Backend.AI, successfully automating the labeling of various systems integrated into a monolith. During this process, I identified the benefit of linking labels from issues to PRs. Unable to find an existing action, I took the initiative to create the auto-label-in-issue action, learning the GitHub API and action scripting in the process.

## Finishing my Internship
This internship has been a significant learning experience for me. Although it's my second internship, it's my first within an IT company, as my previous employer wasn't in this sector. What draws me to Lablup is its commitment to open-source products and the active contributions to the community. The company's horizontal structure allowed me to freely share my thoughts, making me wonder if a company could really operate so collaboratively. One of the greatest aspects of Lablup is the autonomy it offers, allowing you to pursue what you're passionate about rather than what you're obliged to do.

Having developed a strong understanding of the project, I was disappointed when the internship neared its conclusion. Fortunately, Lablup offered me the opportunity to extend my internship, which I accepted, and continued to work on action issues. As there are few developers specializing in this area, I recently presented on the subject at GDG Daejeon. This led to the amusing nickname "Action Mask" from my colleagues.

 ![](https://cdn.lablup.com/202441346_27e360f9_6397_4c79_960d_c47732f5d070_ea1a7af4ce.png) 

I have shared my internship experience with the Open Source Contribution Academy, where I performed well. I believe that my experience at the company was the foundation for that.

A 2022 summer internship at Lablup that turned into a full-time role, and the work and the people that made the author decide to stay.

Recap of my Lablup Internship, Summer 2022

Yujung Kim



## Application Process

I had the opportunity to apply for an internship at Lablup through the **2021 Open Source Contribution Academy**.

 ![](https://cdn.lablup.com/2022_02_16_internship_review1_13c0576594.png) 

Backend.AI was a large-scale and complex project, so the two-month program felt short, and I wanted to learn more about it. Despite my many shortcomings, I wondered, "Is there anything I can contribute?" However, I thought it was a great opportunity to experience real-world work, so I decided to apply. I learned a lot from my mentors while participating in the project, and they were all wonderful people with whom I wanted to continue working.


After passing the document screening, I had my first technical interview. I wasn't sure what questions to expect, so I studied a bit of everything, which I think was counterproductive. Since I had mainly done web front-end development, they asked me about web and front-end knowledge, but I couldn't answer many of the questions. I just did my best to respond. Fortunately, I received a call saying I had passed! Just four days later, I started my internship. The entire process, from the acceptance notice to the interview, moved very quickly.

## Orientation

At Lablup, they develop and manage a solution called Backend.AI, so during orientation, I completed tasks involving installing and using the platform. Most of the tasks focused on developing machine learning models using Backend.AI. Having never studied machine learning before, I felt overwhelmed, but I managed to complete them somehow. During the internship, the most important lesson I learned was: "If you keep searching and trying, things eventually work out." When I didn't know something, I would ask for help; when an error occurred, I would fix it; and before I knew it, I had completed the task.

Here is a brief summary of the tasks I completed during orientation:

* **Task 1.** Change MNIST code to use Fashion MNIST
* **Task 2.** Increase accuracy without changing the model
* **Task 3.** Install Backend.AI
* **Task 4.** Run code on Backend.AI CLI
* **Task 5.** Run code on Backend.AI Web UI
* **Task 6.** Increase accuracy without changing the data
* **Task 7.** Increase accuracy within the time limit

Personally, Task 2 was the most challenging. I tried to increase accuracy using data augmentation techniques, but it didn't work as expected. I added flipped images to train the model, but strangely, adding vertically flipped images actually decreased the accuracy. The accuracy was highest when I only added horizontally flipped images, so I submitted the assignment with that approach.

## Work

Backend.AI is implemented through the interaction of various components.

There are eight components in total: manager, agent, webui, common, client-py, kernels, storage-proxy, and webserver. During my internship, I resolved issues related to the manager, agent, webui, and common components. I'll summarize what I accomplished for each component.

### Manager

This component provides APIs and is responsible for monitoring computing resources and scheduling sessions. As a core service, it handles many functions and contains extensive code. Understanding the code was challenging because the manager connects to all other components, but it was very helpful for grasping the overall architecture.

#### Adding a command (delete old records, free up disk space)

PR (Merged): [https://github.com/lablup/backend.ai-manager/pull/498](https://github.com/lablup/backend.ai-manager/pull/498)

* **Issue Description**
    Session information accumulates in the `kernels` table of the database. This data needs to be deleted periodically, and a PostgreSQL `VACUUM` operation must be performed to free up disk space.
    → Add an `mgr clear-history` command to easily perform this task.

* **Process & What I Learned**
    I used a Python package called Click to create the CLI command and added two options to allow for more detailed commands.
    However, when I ran the command, I encountered the following error during the vacuum operation:

    ```VACUUM cannot run inside a transaction block```

    I tried all the solutions from Stack Overflow and various other sites, but couldn't solve it.
    In the end, the official documentation provided the answer. Commands like `CREATE DATABASE` or `VACUUM` cannot be executed inside a transaction, so the mode had to be changed to AUTOCOMMIT. I solved it using the method below:

    `conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)`

* Reference: [https://www.psycopg.org/docs/extensions.html#isolation-level-constants](https://www.psycopg.org/docs/extensions.html#isolation-level-constants)

#### Moving the location of the maximum container count check

PR: [https://github.com/lablup/backend.ai-manager/pull/504](https://github.com/lablup/backend.ai-manager/pull/504)

* **Issue Description**
    When creating a session, users can set conditions such as the session's name, type, resources, cluster mode, and cluster size. When such a session request is received, the manager must verify that the requirements are appropriate.
    → Move the check that verifies whether the **maximum number of containers is exceeded** to a more appropriate location.

* **Process & What I Learned**
    Conditions that might change regarding whether a session can run after waiting in a queue (e.g., after another session finishes) are checked in `predicates`. This check is re-run every few seconds to determine if the session has become runnable. However, if runnability is determined solely by policy options, regardless of whether other sessions have finished, there's no need to wait in the queue. Therefore, it's better to perform policy validation before placing the session in the queue.
    The check for "whether the number of containers requested by the user exceeds the maximum number of containers" falls into the latter category. Therefore, I moved its location so the check is performed before the session is placed in the queue.

#### Adding a feature to change resource groups

PR: [https://github.com/lablup/backend.ai-manager/pull/511](https://github.com/lablup/backend.ai-manager/pull/511)

* **Issue Description**
    This issue involved adding a feature to change the resource group in the database's `agents` table.

* **Process & What I Learned**
    I used Graphene, a library that provides tools for implementing GraphQL APIs. I added code to the existing `ModifyAgent` mutation so it could also handle resource group change requests.
    However, there was one problem. Approximately every 3 seconds, the resource group's configuration value is read from a TOML file and used to update the database, so any changes made through the mutation would be overwritten. Therefore, in addition to changing the database's resource group value through the Agent mutation, the value in `agent.toml` (the configuration file) also had to be updated.
    → I addressed this by adding an RPC method to the agent component that changes the resource group value in the TOML file, and then calling that method from the manager. More details below!
    This was the last issue I worked on. To add this feature, the webui, manager, and agent components all needed updates. In webui, UI elements had to be added to send the change request, and the manager and agent had to be implemented to handle the request appropriately. I couldn't work on the webui, but I completed the manager and agent components and submitted PRs for them. (I plan to work on the webui once the PRs are merged.)

### Agent

This component manages agent containers, and I had minimal work to do on it. Near the end of my internship, I began working on it to add the feature mentioned above.

#### Adding a resource group change feature

PR: [https://github.com/lablup/backend.ai-agent/pull/327](https://github.com/lablup/backend.ai-agent/pull/327)

* **Issue Description**
    I needed to add an RPC method to change the resource group in the TOML configuration file.
    The `ModifyAgent` mutation will call this RPC method to change the resource group setting in the TOML file before updating the database.

* **Process & What I Learned**
    This was my first time using the `tomlkit` library, which helps you modify values while preserving human-added comments. There wasn't much documentation about `tomlkit`, so I encountered some difficulties. I needed to read the file using its path, modify it, and save it again, but the official documentation only explained `parse`, which creates a TOMLDocument object from a String. However, while using `parse`, I encountered an error and by examining the source files, I found the `api.py` file in `tomlkit`. I learned how to use it by studying the functions defined in that file.

    Finally, I added the RPC method as follows:

    ```python
    async def update_scaling_group(self, scaling_group):
        cfg_src_path = config.find_config_file('agent')
        with open(cfg_src_path, 'r') as f:
            data = tomlkit.load(f)
        data['agent']['scaling-group'] = scaling_group
        with open(cfg_src_path, 'w') as f:
            tomlkit.dump(data, f)
        self.local_config['agent']['scaling-group'] = scaling_group
        log.info('rpc::update_scaling_group()')
    ```

    I opened the file with read permission and used `load` to create a TOMLDocument object called `data`. Then I opened the file in write mode to save the modified `data`.
    I initially wanted to avoid separating the read ('r') and write ('w') permissions. However, when I tried using 'r+' permission, the modified data was appended to the bottom of the original content instead of overwriting it, resulting in duplicated content. The separate approach was the only solution.
    I learned a great deal about `tomlkit` while working on this issue. Since there isn't much information available online, I think it would be valuable to write a separate post about it.

### Common

#### Updating the TimeDuration Class

PR (Merged): [https://github.com/lablup/backend.ai-common/pull/99](https://github.com/lablup/backend.ai-common/pull/99)

* **Issue Description**
    This issue is related to the PR for adding a command in the manager. The `TimeDuration` class returns the difference between two datetime objects. The `clear_history()` function for `mgr clear-history` uses `TimeDuration` to calculate the `expiration_date` and deletes data prior to that date.
    However, `TimeDuration` didn't support `years` and `months` parameters. Initially, I wrote it like this:

    ```python
    unit = retention[-2:]
    today = datetime.now()
    if unit == 'yr':
        yr = int(retention[:-2])
        expiration_date = today - relativedelta(years=yr)
    elif unit == 'mo':
        mo = int(retention[:-2])
        expiration_date = today - relativedelta(months=mo)
    else:
        duration = TimeDuration()
        expiration_date = today - duration.check_and_return(retention)
    ```
    If `TimeDuration` supported months and years, the code could be written more concisely like this:
    ```python
    today = datetime.now()
    duration = TimeDuration()
    expiration = today - duration.check_and_return(retention)
    expiration_date = expiration.strftime('%Y-%m-%d %H:%M:%S')
    ```

    → Update `TimeDuration` to support months and years.

* **Process & What I Learned**
    `timedelta` doesn't have `years` and `months` arguments, so I had to use `dateutil.relativedelta.relativedelta`. `dateutil` is an extension package for the `datetime` module.
    Initially, I tried to unify the return type, but `relativedelta` doesn't support float-type arguments, so it failed the GitHub Action tests. Therefore, I only used `relativedelta` for years and months. I also wrote test code for the first time:

    ```python
    date = datetime(2020, 2, 29)
    ...
    assert iv.check('1yr') == relativedelta(years=1)
    assert iv.check('1mo') == relativedelta(months=1)
    assert date + iv.check('4yr') == date + relativedelta(years=4)
    ```
    `iv` is a `TimeDuration` object. This test code verifies that it returns the desired value when '1yr' and '1mo' are passed. It also tests whether the calculation is correct for leap years.

### Webui

#### Fixing the issue where the Terms/Privacy Policy window doesn't reopen after being closed

PR (Merged): [https://github.com/lablup/backend.ai-webui/pull/1160](https://github.com/lablup/backend.ai-webui/pull/1160)

* **Issue Description**
    As shown below, after closing the Terms of Service or Privacy Policy window by clicking the 'x' button, it wouldn't reopen when the button was clicked again.

![](https://cdn.lablup.com/not_working_after_click_x_button_d9b2dc8e1f.gif)

    There were two buttons to close the window: 'x' and 'dismiss'. If closed using the dismiss button, the window would reopen properly. However, this wasn't the case for the x button.
    → Modify the functionality so that the window reopens even after being closed with the 'x' button.

* **Process & What I Learned**
    Backend.AI webui uses Lit, a library for web components.
    The Terms/Privacy Policy window had a structure where a `backend-ai-dialog` component was nested inside a `lablup-terms-of-service` component. The dismiss button exists within the `lablup-terms-of-service` component, while the x button exists within the `backend-ai-dialog`.
    → Condition 1: When the x button is clicked, both components must close.
    The window only opens when the `show` variable is `false`.
    → Condition 2: When the window closes, the `show` variable must be changed to `false`.
    When closing `backend-ai-dialog`, I dispatched a "dialog-closed" `CustomEvent`. Then I added an `EventListener` for it in the `lablup-terms-of-service` component. In the callback function, I added code to satisfy both conditions above.

#### Fixing the problem of overlapping items in a table

PR (Merged): [https://github.com/lablup/backend.ai-webui/pull/1170](https://github.com/lablup/backend.ai-webui/pull/1170)

* **Issue Description**
    When a session's GPU value was three or more digits, it overlapped with the mount folder item on the right, as shown in the image below.


 ![](https://cdn.lablup.com/session_0d92aabac9.png) 



* **Process & What I Learned**
    Initially, I considered solving this by making the GPU item's width increase dynamically. However, there were parts that didn't work as expected, so I asked for advice. My colleagues helped me brainstorm solutions to the problem, and through that conversation, I was able to find the right direction. Originally, I was so focused on fixing issues quickly that if I thought of a solution, I would immediately try to implement it without considering other approaches. However, while working on this issue, I realized the importance of taking time to view problems from the user's perspective and consider whether there might be better solutions.
    In the end, I solved the problem by changing the layout of the items as shown below, grouping related elements together.

  ![](https://cdn.lablup.com/batch_08dc526763.png) 


## Company Culture

To summarize in one phrase: it was very free and horizontal. Everyone is addressed by their name with "nim" attached (even the CEO!).

* **Work distribution is flexible**
    You don't exclusively do front-end or back-end work. You can experience a variety of tasks, which was the best part for me. This could be seen as both a pro and a con, but for me, it was definitely a pro.

* **Flexible work hours + remote work**
    Working at the office twice a week was recommended, but this seemed to change depending on the COVID-19 situation. Arrival times are flexible, and departure times are also flexible. However, everyone works hard to match that freedom.

* **Official language is English**
    There are foreign employees, so English is used in general meetings and for GitHub Issues & PRs. In company channels, if everyone in the conversation is Korean, you can write in Korean, but otherwise, you write in English.

* **Lunch provided, gym subsidy, liberal vacation policy, use all your legal annual leave**
    Basically, if you work at the office, lunch is provided and we eat together. Of course, if you have a lunch appointment, you can eat separately. A portion of gym fees is subsidized, but I didn't use this benefit so I'm not sure about the details. The atmosphere regarding vacation was really liberal—you just have to give advance notice. Equipment is provided for each person. During the internship, I received a MacBook, mouse, and keyboard to use.

* **When you don't know something, ask! Everyone is kind and explains things well**
    The hardest thing for me during the internship was asking questions. I was worried that colleagues would think I didn't know the basics, so I spent a lot of time researching and exploring on my own. As time went by, I realized the importance of asking questions. No one responds negatively—everyone explains things so well that I learned something new every time I asked. When I was struggling with an error, they would help me resolve it.

Two months is a short time to understand an entire company culture, but I've summarized what I experienced during that period. I had never considered working at a startup before, but my perspective changed significantly while working as an intern at Lablup. Everyone was constantly striving to create a positive company culture.

## Review

When I started the internship, my mind was filled with worries. However, as I began to understand things step by step while resolving issues, it became increasingly enjoyable. I think I wrote that I wanted to work passionately with good people when asked about my motivation for applying, and it was exactly as I had hoped. Until then, I had only done front-end development, but I was able to experience back-end work. It was much more interesting than I had expected, so I plan to study more back-end development in the future.

Real-world work was definitely different from what I had experienced before. On GitHub, I had only used the code and pull request tabs, but I used Actions for the first time. I was able to learn how projects are managed and began using git commands more diversely. My perspective broadened significantly while working here, and I believe what I've learned will be very helpful in the future.

It was such an enjoyable and beneficial experience that the two months felt short. It's unfortunate that it has to end, but I plan to study hard so that I can meet them again as an improved version of myself.

A first internship at the startup Lablup, two months of hands-on backend work and a flat team culture that helped the author grow as a developer.

My First Internship Experience at an IT Startup, Lablup

Chrysan Angela

I had the opportunity to learn valuable lessons on AI, data science, computer vision, container technologies, and CLI, as well as working with experienced professionals and new friends.

The word “summer” in my article and the cold, early winter breeze outside seem to highlight the fact that this blog post has been long overdue. I started my internship on June 28, exactly 5 months ago, by the time this article is published, and finished it on August 20. I am finally paying the article debt I owed.


### How It Started
Lablup is a startup based in Seoul, South Korea. Their primary product, Backend.ai, is a “streamlined, container-based computing cluster orchestrator that hosts diverse programming languages and popular computing/ML frameworks, with pluggable heterogeneous accelerator support including CUDA and ROCM”. I quoted the definition from their open-source Github repository, which can be accessed through this [link](https://github.com/lablup/backend.ai)

<figure>
 <img src="https://cdn.lablup.com/Lablup_Logo_with_text_horizontal_text_with_color_L_60ba8caaa1.png" alt="Lablup_Logo">
 <figcaption>The Lablup logo</figcaption>


I first heard about Lablup Inc. when one of my seniors interned there. At that time, I just finished my third semester, so I barely fulfilled any of the qualifications needed to apply for computer science internships. Last semester, I felt ready to try applying to the internship program. What interested me the most about the internship was how I would be able to practice my AI and software/web development skills at the same time. I applied through their internal application opening, as the CUop (company-university partnership program to facilitate internships for juniors and seniors) program stopped accepting international students due to visa issues. I felt a bit anxious since I didn’t have a strong background in backend, but fortunately, I got accepted :)


### The First Few Days
The program is conducted mostly remotely because of the COVID-19 pandemic (ㅠㅠ), but we could still come to the office a few times a week, provided we filled in the “office attendance list” beforehand. It wasn’t required, but I moved to Seoul from Daejeon for the summer for a more immersive internship experience (+a valid excuse to spend more time in Seoul). I spent the first few days after moving in adjusting to the busy city life in Seoul, exploring the much richer mix of cultures in the capital city.

<figure>
 <img src="https://cdn.lablup.com/street_b922647287.png" alt="Backend.AI Architecture">
 <figcaption>Some of my favorite pictures from my summer stay—do you recognize any of them?</figcaption>
</figure>

On the first internship day, we were encouraged to come to the office in person for a brief orientation. There were six interns in total; everyone ended up coming to the offline meeting. Quite a lot of the full-time members came to the orientation too. Sadly, this was also going to be the last time all interns got to “assemble” together (._.) We listened to a short overview of the internship program and a presentation about the history of Lablup Inc., and then had lunch outside in groups of four. We were given instructions on how to join the Teams chat and other internal collaboration platforms, and then we were discharged early.

The orientation tasks consisted of installing Backend.ai on our local machine, launching the web version of Backend.ai, and experimenting with neural networks to solve a Fashion-MNIST classification task based on TensorFlow’s tutorial. After we were finished, we submitted screenshots of our work to Lablup’s internal platform for archive. (P.S. Backend.ai is available for free for everyone, so check their page out if you’re interested!)

As a side note, at the time this task was assigned, Backend.ai was yet to support ARM-based architectures (M1 Macs being one of them). I unknowingly tried to install Backend.ai on my M1 Mac, and the result was a disaster.. I ran into so many errors, even though the installation was supposed to work perfectly smoothly on Intel-based Macs.

<figure>
 <img src="https://cdn.lablup.com/orientation_tasks_3ef65889c6.png" alt="Backend.AI Architecture">
 <figcaption>The orientation tasks for week 1</figcaption>
</figure>


### Research Team Tasks
Starting from the second week, we were introduced to tasks from the research team. Over the course of the internship, we attempted to analyze the data from two open-source papers on face coverings: Efficacy of masks and face coverings in controlling outward aerosol particle emission from expiratory activities by Asadi et al (Scientific Reports, 2020) and Face coverings and respiratory tract droplet dispersion by Bandiera et al (The Royal Society Open Science, 2020). It took me a week to finish the task from the first paper. The #1 lesson I learned was that data visualization is only a very small part of data analysis; much of the work is spent on structuring the dataframe and cleaning raw data. I used Jupyter notebooks on Backend.ai cloud to work on this task, with the help of Pandas, NumPy, Matplotlib, and Seaborn.

 <img src="https://cdn.lablup.com/final_results_cb44a7cb9e.png" alt="The final results I got for the first task">
 <figcaption>The final results I got for the first task</figcaption>
</figure> 

The second paper was much more challenging than the first one, primarily because we had to work with a large set of image data as opposed to text data. I was supposed to count the number of particles deposited using particle detection, but I still couldn’t find a good algorithm by the end of the internship period. For this task, I experimented with an ensemble of OpenCV functions.

<figure>
 <img src="https://cdn.lablup.com/Two_snapshots_35cb3ec437.png" alt="Two snapshots of the images that we were supposed to process">
 <figcaption>Two snapshots of the images that we were supposed to process</figcaption>
</figure>

### Backend.ai Issues
Aside from the research team tasks, which mostly deal with data science, we were also working on resolving issues in the Backend.ai repository — this is arguably the main part of the internship! It took me quite a long time to understand the codebase at first, as there were quite a lot of components and this is the first time I got to study an industry-level repository. As a result, I wasn’t able to work on as many issues as I had previously expected…

To prepare for my tasks, I learned more about Linux terminal commands, shell scripts, Docker containers, and client-server interaction. At that time, Backend.ai was about to have a major update: new support for ARM, and more consistent command conventions. A friend worked on the former, while I collaborated with another friend for the latter. It was such a memorable experience to be able to learn from friends and mentors; it reminds me that I have so much more to learn! By the end of the internship, I made the following merges to the Backend.ai repositories:

 ![](https://cdn.lablup.com/10_1_backend_AI_ac404b7f94.jpg) 
 ![](https://cdn.lablup.com/10_3_backend_AI_4a3960b086.jpg) 


### Company Culture
Frankly, I felt a bit apprehensive at first, since I’m not very fluent in Korean. However, everyone is friendly and English is the official language during meetings, so I felt very welcomed (🥺). Although I tried to solve tasks on my own as much as I could, I still needed help from the full-time members from time to time. Everyone was busy, so I felt a bit hesitant to ask, but they still tried their best to help amidst their schedule.

Every morning at 10 AM, we joined a short 15-minute meeting where important updates and schedules were briefed to all members. Besides the daily morning meeting, we also had other company “seminars”. Every Tuesday, we had a “codebase study”, where interns and new members took turns presenting a certain part of Backend.ai. It helped me understand how Backend.ai works more quickly, as there was a lot to take at once. Every Wednesday, we had “pebble seminar”, where we discussed recent trends in the AI/ML/data science field. Every Friday, we had an OKR meeting to present our progress for the week, as well as get feedback from mentors. At the end of the fourth week, we had a midterm evaluation instead, and in the final Friday, we had a final evaluation presentation.

Aside from those, we also had a cultural day where we got to play games together. It had to be done online, unfortunately, because of the pandemic. We played in teams of “equal strengths” (aka average experience at Lablup). Also, there were a lot of prizes :D. I really enjoyed the chance to interact in a more informal setting with other members.

<figure>
 <img src="https://cdn.lablup.com/workspace_c58fd8ac9b.jpg" alt="My workspace">
 <figcaption> My workspace at Lablup, sadly this is the only picture I took in the office :(</figcaption>
</figure>


### Closing Remarks
A few days after the official end of the internship, one kind unnie took some of us for a treat (😊). We had a Western-style (?) platter and gelato. Everything was so good! At the same time, I genuinely felt sad for having to say goodbye to the friends whom I had worked closely for the past two months.

<figure>
 <img src="https://cdn.lablup.com/drools_47cc27f779.png" alt="drools">
 <figcaption>drools</figcaption>
</figure>


All in all, my summer this year has been truly exciting, thanks to Lablup’s internship and the experience it gave me. On top of the technical lessons, I also got a chance to explore Seoul, experience a taste of working-class life, and make new friends. If there’s one thing I would have done differently, it is that I should have written this blog post earlier..

A summer internship at Lablup spanning AI, data science, computer vision, and container technology, and the hands-on experience that came with it.

My Summer Internship at Lablup

Sami Hammami

I joined Lablup in April 2021. It’s now been eight months, and I’d like to briefly share my experience so far.

---

### Application Process

Before joining Lablup, I worked at a finance-related company in Yeouido. It was a great experience where I met some wonderful people. One of them later moved to Lablup and shared positive things about the company, which sparked my interest. I decided to give it a try and submitted my resume. Shortly after, I was contacted and completed a programming test followed by an interview.

If you’re interested in our job openings, feel free to visit Lablup’s regularly updated [homepage](https://www.lablup.com/careers).

---

### Background

While I was on a scholarship in France, I had the opportunity to study abroad for a semester. At the time, I didn’t know much about Korea or its culture, but I had always been fascinated by Asia due to my early interest in Japanese video games and animation.

In 2016, my university established a partnership with Seoul National University of Science and Technology. On the day of the announcement, my professor asked who would be interested in spending six months in Korea. Without much thought, my hand went up—almost involuntarily.

It was my first time living in a country so culturally different from my own. I met amazing people, built strong friendships, and learned a lot—about both human relationships and technology. The experience was incredibly enriching. But as with most good things, it eventually came to an end.

As graduation approached, I began to think seriously about my next steps. Many of my classmates were applying to companies in France, and some had already received offers. But I didn’t feel that path was right for me. I wanted something different—something that would allow me to return to Korea and continue exploring new challenges.

So, I started researching visa procedures, housing options, and job opportunities. After graduation, I booked a flight to Seoul and began my job search.

One of my master's majors was cybersecurity, and I hoped to gain experience in that area. Soon after, I found a finance company that needed help with both web development and security. During my time there, I worked with brilliant people who generously shared their knowledge—one of whom eventually joined Lablup.

He told me about Lablup’s open-source and AI research projects, its wide-ranging technologies in cloud computing, and the company’s rapid growth. This piqued my interest, and around the same time, Lablup began hiring more engineers and developers. I submitted my application, passed the resume screening and programming test, and proceeded to the interview.

---

### About Lablup

If you're passionate about cloud computing, software, or AI, Lablup is one of the most exciting tech startups in Korea. Everyone here is both passionate and highly skilled.

Lablup develops and manages [Backend.AI](https://www.backend.ai), a resource orchestration platform for AI research. More specifically, Backend.AI is a container-based computing cluster orchestrator that hosts various programming languages and machine learning frameworks. It allocates and isolates computing resources across multiple sessions.

During my first week, my task was to install and run Backend.AI on my local machine and create a use case using similar tools. One of these involved building a chat application using `aiohttp`, an asynchronous HTTP client/server. The whole process went smoothly. Everyone was open to questions and eager to help, which made the experience enjoyable and left a strong first impression.

---

### Experience

In my first week, I focused on simple tasks and resolved beginner-level issues in Backend.AI to better understand the platform’s architecture and the development workflow.
Backend.AI has a complex structure, so it took some time to fully grasp. During this time, I worked on resolving "good first issues."

Once I understood how the system’s components interact, I shared my insights into the domain, group, user, and policy management systems through a presentation that explored function calls, data storage, and logic.

---

### License Management

Although the Backend.AI platform is open source, companies or individuals must purchase a license to receive engineering support or to access the full range of features.

At first glance, the concept of licensing may seem simple—users buy a license to access the software. But the actual system is more involved. It includes generating encrypted keys for each user, creating customized license keys, and having the system validate these keys via request handling, expiration updates, and more.
There’s quite a bit of work for both humans and machines.

To streamline this, I’ve mainly been working on integrating the license server into the back office to provide a clearer view of and easier access to license management.

Through this project, I not only became familiar with Golang (a programming language developed by Google), but also deepened my understanding of AWS cloud instances and security group configuration.
My time was divided between studying and implementation. I frequently referred to both internal and external resources during work hours and benefitted greatly from feedback during code reviews or casual chats with colleagues over coffee.

---

### In Closing

As an orchestrator, Backend.AI is composed of many components—Manager, Agent, Storage Proxy, and more.
Working at Lablup means dealing with a variety of tools, programming languages, and systems, and developing a deep understanding of how they all fit together.

I often wonder what I’ll be doing in five years. It’s a tough question, and I know not all hopes come to fruition. But I sincerely hope Backend.AI becomes even more widely adopted, and I plan to continue contributing to it—alongside other exciting projects.

I’ll continue working hard with a genuine passion for programming, striving to become a vital part of the company and a supportive colleague to my teammates.

If you’re passionate about programming, AI, cloud computing, or IT in general—and want to work with other passionate, talented people—consider joining Lablup.

Thank you for reading.


Eight months at Lablup, a candid look at growing in AI and the goals that came into focus along the way.

Working at Lablup for 8 months

Taewan Park

## Internship Application Process

The first semester of my junior year was coming to an end, and it was time to start preparing for my graduation project for the upcoming semester. While contemplating meaningful activities for the summer vacation, I came across a public internship recruitment post from Lablup on Facebook.

After reading the job posting, I thought it was an opportunity I couldn't miss. It was one of the startups I was already familiar with, and I felt their internship program was much more substantial compared to the work-study programs offered in collaboration with my university. I immediately began filling out the application. The application contained many questions, mainly focused on my programming and project experience.

Some time after submitting my application, I received an email for an interview. I was happy to have passed the initial screening, but I was also very nervous since it was my first technical interview. After agonizing over what to prepare, I ended up not preparing much for the technical aspects and instead focused on reviewing the programming and project experiences I had written about in my application before joining the video conference.

Fortunately, the interviewers asked extensively about my programming and project experience. They did ask some computer science technical questions, but I couldn't answer many of them. To be honest, I was so flustered at the time that my head was spinning, and I honestly thought I was going to faint. (Looking back now, I wonder how I even passed.)

A few weeks after the interview, as I was finishing up the semester and making plans for summer vacation, I received an acceptance email, which made me ecstatic!! I chose to start my internship a week later and spent that week having fun.

## Internship Tasks and Daily Life

The Lablup internship started with a one-week bootcamp assignment where we tackled basic tasks from various fields within the company. For the remaining weeks, we were assigned tasks and issues in areas we wanted to explore further or were particularly interested in.

Lablup's core platform is open source! Therefore, the internship assignments were also carried out publicly on GitHub.

## Bootcamp Assignment

The bootcamp assignments can be summarized as follows:

* Learn how to use Backend.AI
* Train machine learning and deep learning models using Backend.AI
* Install Backend.AI in a local development environment

Backend.AI is Lablup's core product. Since I was scheduled to work on actual Backend.AI development tasks after the bootcamp, I had to complete these assignments thoroughly before moving on.

> ❗ **Note**: Clicking on each image will take you to the corresponding PR.

## Backend.AI Webserver

[![Webserver ping API PR](https://cdn.lablup.com/02_add_ping_api_e728b31a50.png)](https://github.com/lablup/backend.ai-webserver/pull/38)

After the bootcamp, this was the first "good first issue" I tackled while diligently studying the Backend.AI source code. The task was to add a `ping` API to check if the web server was running correctly. Since Backend.AI's web server uses `aiohttp`, I wrote the code as follows, and my pull request was approved without any additional revisions.

## Backend.AI Agent

[![PR review](https://cdn.lablup.com/03_add_path_to_default_3f56966d54.png)](https://github.com/lablup/backend.ai-agent/pull/296)

Next, I took on an issue with the agent. The Backend.AI agent is installed on computation nodes and is responsible for managing and controlling containers. This task required ensuring that the `~/.local/bin` path was always included in the container's `PATH` environment variable when a container is created by the Backend.AI agent. I was initially overwhelmed by the vast amount of code in the agent and didn't know where to start, but our CTO, Jun-ki Kim, gave me a hint to examine the code that executes when the `ai.backend.kernel`'s `BaseRunner` object is created, which helped me solve it successfully!

P.S. While working on this task, I learned that the `~/.local` folder, like the `/usr/local` folder, is used for installing local or third-party packages. However, while `/usr/local` is for all users, `~/.local` is for a single user. Notably, pip uses it...! (And of course, `bin` contains the executables).

## Backend.AI Manager

[![## Backend.AI Manager](https://cdn.lablup.com/04_duplicate_target_dir_check_e0bd4697c3.png)](https://github.com/lablup/backend.ai-manager/pull/461)

The next task arose while I was reviewing this pull request. The PR was for adding a feature to the web UI that allowed creating sessions by setting an alias for a virtual folder. During the review, I discovered a bug that allowed duplicate alias names to be entered. If a duplicate alias name is entered in the web UI, an error occurs when the Backend.AI agent actually creates the session because you can't have two folders with the same name in the same directory, preventing it from working correctly. Although this issue was patched and the PR was successfully merged, there was consensus that it would be beneficial to implement safe coding practices in the Backend.AI manager as well, so I took on this task. While working on it, I not only implemented the folder name duplication check but also cleaned up some duplicated code from backporting, submitted the PR, and it was successfully merged.

## Backend.AI Test

[![add_new_cli_based_test](https://cdn.lablup.com/05_add_new_cli_based_test_35de5929f0.png)](https://github.com/lablup/backend.ai-test/pull/1)

The final task was special because I collaborated with another intern. The other intern refactored the Backend.AI CLI commands, and my role was to write integration tests using the refactored CLI commands. I wrote test cases by executing commands that included CRUD operations (Create-Read-Update-Delete) for each command and verifying that the correct values or results were returned. It took longer than expected because there were more commands than I anticipated, but it was the most rewarding task. It was a unique experience to write different code concurrently based on code committed by another person.

## OKR Meeting

<figure>
 <img src="https://cdn.lablup.com/06_okr_tracker_1_f2c12fcd30.png" alt="OKR Meeting">
 <figcaption>OKR Tracker</figcaption>
</figure>

Lablup interns are required to attend an OKR meeting once a week. An OKR meeting is where we present a summary of what we accomplished that week, what challenges we faced, and how we resolved them. After the presentation, the Lablup team members provide feedback on each task. Receiving feedback helped me fill gaps in my knowledge, and listening to the other interns' presentations was beneficial as I could learn about other areas I wasn't directly involved in.

## Internship Review

The company is located a 5-minute walk from Seonjeongneung Station. The office isn't very large, but it has various amenities. There's an Xbox and an electronic piano for stress relief. (The Xbox has an Ultimate Game Pass subscription!) You can come and play whenever you can't focus on work or need a break.

The company has a very relaxed atmosphere and promotes horizontal communication. Everyone is very kind and answers questions well. Thanks to this culture, I think I was able to adapt well without being too shy initially.

Every morning at 10 AM, Lablup has an all-hands meeting to briefly track others' work and achievements and share our plans for the day. Also, anyone can participate in any meeting held at Lablup. I think this is a great culture because interns can also participate in various meetings and build practical knowledge.

As I performed various tasks during the internship, I felt that I was getting closer to real-world work. Although I still have a long way to go, I was proud to have contributed to the company's product in various areas, starting with simple issues. I encountered many problems while solving practical tasks, but I think I learned a lot through the process of trial and error. Also, the Backend.AI codebase was the largest and most complex project I had ever worked on, and figuring out how it worked by analyzing it step-by-step gave me a great sense of accomplishment.

I wanted to gain diverse experience by going to the office and working alongside my mentor, but unfortunately, as soon as the internship started, the social distancing level was raised to Level 4, so I was disappointed that I couldn't visit the office much.

A 2021 summer internship at Lablup, spent contributing to open-source development, with a close look at the projects and the work behind them.

2021 Summer Lablup Internship Review

Lee yongkyun

I interned as a software engineer at Lablup, a Korean startup, for seven weeks. After my Goldman Sachs internship ended, I had a lot of free time before the fall semester began. I started working with the mindset of "I might as well do something productive" and it turned out to be a valuable experience.

### Internship Application Process
Lablup was a mentor company for Contributhon, operated by [OSS](https://www.oss.kr). Contributhon is a competition where participants are matched with companies to contribute to their open-source projects for six weeks. I briefly considered participating in this competition but wanted to work full-time, so I contacted Lablup's founder/CEO, interviewed, and started working.

### About Lablup
[Lablup](https://www.lablup.com) is one of Korea's leading tech startups, developing Backend.AI, a resource management platform for AI research, as an open-source project.

While Google Colab is convenient and useful for simple tasks, it has limitations. For example, its limitations become apparent when using programs that only run on specific OSes or when dealing with large datasets. Therefore, institutions conducting serious AI research typically don't rely on Google Colab. When I was at Naver, we used an internal platform called NSML, and at Onclusive, we ran AWS instances to train models to ensure sufficient GPU usage. Although it was a straightforward method, it was quite cumbersome as I had to download necessary programs each time, transfer local files via scp, and then stop the instance after use.

Lablup earns revenue by providing B2B technical support for Backend.AI to various institutions. Around the time I started my internship, they launched a service targeting general consumers, similar to Google Colab. It's more versatile and useful than Google Colab but is a paid service.

### Internship Experience
Instead of working on one large project for seven weeks, I resolved various issues. Backend.AI is very complex, so it took several days just to set it up and get it running on my personal computer, and about a month to understand the system's workflow. To run everything locally, including the UI, I had to have six programs running. Working with six terminals open simultaneously was actually pretty cool, though. Anyway, the work I did as an intern is as follows:

#### Even Allocation of GPU
Some research assumes that GPUs are evenly distributed when using multiple GPUs for model training. Therefore, we decided to develop an algorithm instead of a simple greedy method. For example, if a 0.3 GPU workload needed to be distributed across two 0.2 GPU capable slots, we programmed it to choose allocations of 0.15/0.15 rather than 0.2/0.1.

I spent most of my time developing the algorithm. It felt like solving LeetCode challenges, but I had to consider far more scenarios than typical coding problems, and there were more edge cases than expected, making it difficult to write clean code. I wrote the code to enable this algorithm to function within the system.

Github Link - [PR](https://github.com/lablup/backend.ai-agent/pull/228), [Issue](https://github.com/lablup/backend.ai/issues/85)

#### etcd3 Library Replacement
Backend.AI uses `asyncio` extensively. `python-etcd3` is an open-source library that enables the use of etcd with `asyncio`, but it hadn't been maintained for a long time, and a grpc-related error that occurred when the Backend.AI agent terminated had been unfixed for quite some time. So, I found a library called `aetcd3`, which was a fork of `etcd3aio` (itself a fork of `python-etcd3`), made a few modifications, replaced `python-etcd3` with it, and made Backend.AI run without errors in the local environment. However, replacing an external library in a complex system like Backend.AI requires great caution, so ultimately, my work was not merged during my internship. Nevertheless, by experimenting with various open-source libraries and reading issues posted by people, I gained some insight into the open-source community.

Github Link - [Issue](https://github.com/lablup/backend.ai/issues/189)

#### XFS Storage Proxy
XFS is a file system created by Silicon Graphics. I developed the XFS storage proxy for Backend.AI. Backend.AI uses a virtual folder system. Users can create their own personal virtual folders in the cloud and store files, libraries, or code. Of course, features like folder sharing, duplication, and access control are implemented. Virtual folders are actually stored in a specific structure on a file system designated by the program, and XFS is one of the file systems supported by Backend.AI.

Basic virtual folder functionality was already implemented, so I adapted it for XFS. In XFS, the file system is managed on a per-project basis, with project information stored in `/etc/projid` and `/etc/projects`. So, when new virtual folders were created or deleted, I updated these two files accordingly. Each project has a storage quota, and if the file capacity was exceeded, I manually increased the quota.

Since this was my first experience with a file system like XFS, it took quite some time to understand its working principles and usage. As macOS does not support XFS, I created an Ubuntu instance on AWS to work. Since I had to work exclusively in the terminal, I learned how to use tmux and vim, which turned out to be surprisingly fun.

Github Link - [PR](https://github.com/lablup/backend.ai-storage-proxy/pull/2)

#### Leave Virtual Folder
I implemented a feature allowing users to leave shared virtual folders. When a user leaves a virtual folder shared by multiple people, I applied the changes to the managing database and revoked the user's access rights to that folder. I also enabled this functionality through the Python client. This involved standard backend work, implementing API endpoints and related functions.

Github Link - [Issue](https://github.com/lablup/backend.ai/issues/150), [PR (manager)](https://github.com/lablup/backend.ai-manager/pull/317), [PR (client-py)](https://github.com/lablup/backend.ai-client-py/pull/123)

#### Clone Virtual Folder
The ability to clone a shared or group virtual folder to a personal virtual folder is useful when users want to easily start new projects using existing ones as templates or experiment with the contents of shared virtual folders. Similar to the 'Leave Virtual Folder' task, I implemented the related functionalities in the Backend.AI agent, manager, and Python client.

Github Link - [Issue](https://github.com/lablup/backend.ai/issues/150), [PR (manager)](https://github.com/lablup/backend.ai-manager/pull/323), [PR (client-py)](https://github.com/lablup/backend.ai-client-py/pull/127)

#### Update Installation Script
I updated the installation script that installs all necessary programs to run Backend.AI and configures the environment appropriately. The goal was to make it so that running the script would set up Backend.AI to run immediately without further manual intervention. This included tasks like configuring etcd and PostgreSQL in Docker.

Github Link - [Issue1](https://github.com/lablup/backend.ai/issues/216), [Issue2](https://github.com/lablup/backend.ai/issues/224), [PR](https://github.com/lablup/backend.ai/pull/215)

#### Global Dotfiles
Backend.AI allows users to modify configuration files like `~/.bashrc` or `~/.zshrc` in their respective containers. Since administrators also need to be able to modify global configuration files for group settings, I implemented a feature to modify configuration files like `/etc/bash.bashrc` or `/etc/profile` at the domain or group level.

I enabled storing dotfiles in the PostgreSQL table that manages groups and domains, and added functionality to the Backend.AI manager, agent, and Python client to create, modify, and delete these dotfiles. I also modified the code to copy these dotfiles to the desired location when a kernel (compute session) starts.

 ![](https://cdn.lablup.com/port_validation_screenshot_3bd5033924.png) 
Github Link - [Issue](https://github.com/lablup/backend.ai/issues/210), [PR (manager)](https://github.com/lablup/backend.ai-manager/pull/329), [PR (agent)](https://github.com/lablup/backend.ai-agent/pull/232), [PR (client-py)](https://github.com/lablup/backend.ai-client-py/pull/132)

#### Service Port Validation
Apps like Jupyter Notebook run inside the Backend.AI container using specific ports, and users must enter the service and corresponding port number in advance. The port numbers users enter must satisfy several conditions. For example, the same port number cannot be duplicated, it must be a number between 1 and 65535, and users should not input port numbers used internally by the system. I wrote code to validate the port number entered by the user before the kernel (compute session) starts and raise an error if there's an issue. I also updated the UI to display these errors.

Github Link - [PR (manager)](https://github.com/lablup/backend.ai/issues/150), [PR (agent)](https://github.com/lablup/backend.ai-agent/pull/235), [PR (console)](https://github.com/lablup/backend.ai-console/issues/715), [PR (common)](https://github.com/lablup/backend.ai-common/pull/52)

### Internship Review
The Lablup internship was challenging but enjoyable. Most of the technologies used to run Backend.AI were new to me, so the process of learning them while working was fascinating. The thrill I felt when the code I implemented worked correctly after hours of reading the existing codebase and various online documents was immense. It was meaningful to be able to work on various components of Backend.AI (manager, agent, console, Python client, installation scripts, storage proxy), and I especially learned a lot by getting to see the big picture of how the Backend.AI system operates.

Due to COVID-19, I worked remotely for most of the internship, but I think the company atmosphere was still good. The Lablup founders are exceptionally talented and thoroughly answered any questions I had. I also can't forget the dedication of some team members who would push code at 2:30 AM and then show up for the morning meeting. Working at a tech-focused startup demands more specialized skills and effort than other types of startups, but the sense of accomplishment seems to be correspondingly high. Observing the Lablup team members work so diligently over the years has also raised my own standards for hard work.

A 2020 summer internship at Lablup, seven weeks as a software engineering intern working through technical issues across the Backend.AI platform.

2020 Summer Internship Review

Kangmin Kwon

This post is a review of my internship at **Lablup** from December 22, 2020, to February 19, 2021.

## Internship Application Process

Kookmin University's Department of Software offers internship opportunities with IT companies every vacation. This was during my senior year's winter break, when most students are typically preparing for job hunting.

> **However, I felt strongly that rather than just preparing for employment, I wanted to gain more real industry experience and make a meaningful contribution.**

So, I applied for the internship program during this winter vacation as well.

 ![](https://cdn.lablup.com/kangmin_intern_review_img01_8d2033e40b.png) 

Seeing the Programmers job posting in the picture above, I was drawn to the phrases **"various tasks possible depending on my interests and capabilities"** and **"someone who wants to experience actual model training and data analysis using a machine learning platform,"** so I applied. I had worked on many machine learning-related projects during my previous internship at Common Computer, and I felt I could contribute meaningfully.

After applying for the position, I had an interview with Jonghyun and was accepted.

 ![](https://cdn.lablup.com/kangmin_intern_review_img02_5626f263f3.png) 

## What kind of company is Lablup?

Whenever I do an internship, there's something I always ask the seniors: What does the company develop and what is its business model (BM)? This is because **I want to understand what kind of work my company does and what I'm helping to create.**

> **A company that develops Backend.AI, an open-source resource management platform for AI research.**

In a nutshell, it's a company that eliminates **unnecessary hassles**. \~\~(Once you try it, you'll understand how much of a lifesaver it is)\~\~

When working on AI-related projects, there are many frustrating situations. Especially when preliminary setup is needed to run a model. Things like configuring dependencies required by the model, or manually installing tools to run the model. When using PyTorch or TensorFlow, I had to install various libraries used by the model one by one and manage their versions.

During a previous internship, a senior told me this:

> I believe any problem that takes more than 30 minutes to solve is wasteful. While it's important to learn by working through challenges on your own, not all struggles are productive... If you're stuck for over 30 minutes, just ask me. **And let's document this so others don't have to face the same problem!**

**These frustrating tasks are often just time-wasting obstacles, whether you realize it or not.** Lablup helps with this through its product, **Backend.ai**. Through **pre-built kernel images**, users don't need to go through the aforementioned tedious and frustrating processes; they can just **use** it!

## What Did I Do?

  - **Make Apache-Spark kernel Image**
  - Make Python kernel Image
  - Make Minecraft Bedrock Server Image
  - Docsprint
  - Translate Backend.ai tutorial (English -> Korean)

In summary, these are the tasks I performed. The Python kernel image and Minecraft kernel image were practice for the main task of creating the Apache-Spark image. It was a kind of *tutorial process*, but perhaps due to my lack of understanding, I ran into many issues, especially with **Apache-Spark**, where I remember struggling quite a bit with troubleshooting. For Docsprint, Lablup manages its documents using Sphinx, and since the docs version was quite old, everyone in the company spent 1-3 days organizing the documentation. \~\~(Maybe it's called Docsprint because it's Documentation + Sprint...?)\~\~ There was also a Backend.ai tutorial in PPT format, which I read and practiced with to understand what Backend.ai actually is. In the process, thinking there might be Korean clients, I worked with another intern to translate it into Korean.

## Process of Making a Kernel Image

To create a kernel image, I followed roughly these steps:

1.  Get assigned a task from the project team
2.  Learn about the image for the assigned task
3.  Check necessary dependencies and write a Dockerfile
4.  Test the written Dockerfile on a local Macbook
5.  After testing, add the LABEL at the bottom
6.  Push the completed Dockerfile and service-def to the hub
7.  Test with backend.ai and troubleshoot
8.  PR & Code Review
9.  Merge

## Troubleshooting for Apache-Spark

I'd like to share some challenges I encountered while creating the Spark kernel image. They can be summarized as follows:

1.  Issue where ENV settings in Dockerfile were not applied
2.  Error due to lack of permission for the directory created by `spark-master.sh` to write server logs
3.  API endpoint error

### 1. Issue where ENV settings in Dockerfile were not applied

 ![](https://cdn.lablup.com/kangmin_intern_review_img03_bc695e0496.png) 


The first one was really frustrating. No matter what I did, it didn't work, so I asked for help... and the problem was simply that my backend.ai version was too outdated. It was a bug that had already been fixed, but due to my limited DevOps experience, I didn't realize I needed to pull updates frequently. After doing a `git pull` and resolving version conflicts, it was fixed immediately...

### 2. Permission denied issue

The second issue was that Spark binary files and package libraries were located under `/opt`, but the permissions granted by Backend.ai only allowed read and execute access to `/opt`, not write access.

 ![](https://cdn.lablup.com/kangmin_intern_review_img04_dec30ff6cb.png) 

This makes sense from a security perspective. To solve this, I used a `bootstrap.sh` file to copy the `/opt/spark` directory to `/home/work` when the container started. It wasn't an ideal solution, but it worked (though, as I'll explain later, I didn't ultimately solve it this way and changed this method when I encountered the third error).

### 3. API endpoint error

![](https://cdn.lablup.com/kangmin_intern_review_img05_9a40b93a00.png) 


This problem occurred because the app took time to boot. So I thought, "Why not make it start immediately?\!"

 ![](https://cdn.lablup.com/kangmin_intern_review_img06_fd7cda18b7.png) 


So, I made `start-master.sh` execute as soon as the container started. It ran well without errors, but a senior mentioned that "package binaries don't need to be under the home directory," so I had to fix both issues 2 and 3 together. In other words, I started optimization work.

## Optimizing the Kernel Image

First, if I removed the package binaries from home, I had to remove the copying part in `bootstrap.sh`, which meant issue #2 wouldn't be solved. \~\~(Wait... go back to how it was..?!!)\~\~ **Luckily, there was a solution.** A senior gave me a hint that **for large projects like Spark, there should be a way to specify a separate working directory.** And indeed, there was. I could specify a separate location using `export SPARK_LOG_DIR` in `spark-env.sh`. I added this to `bootstrap.sh` to solve it. This way, Spark could run with the necessary permissions.[^1]

[^1]: https://hoyy.github.io/posts/spark-start-install-standalone



 ![](https://cdn.lablup.com/kangmin_intern_review_img07_f47ac09f67.png) 


Then, I also worked on a task to make PySpark work with Jupyter Notebook. This was easy; after installing it via `pip install` in the Dockerfile, running Jupyter Notebook with backend.ai worked fine, as shown below.

 ![](https://cdn.lablup.com/kangmin_intern_review_img08_06ca94268e.png) 


Finally, leveraging the overlayfs characteristics in Dockerfile, I wrote the Dockerfile to be as similar as possible to the other existing Docker images.

 ![](https://cdn.lablup.com/kangmin_intern_review_img09_5c9c418f9d.png) 


I had to use the common parts from other images as much as possible and defer the differing parts to later stages. This is essential for resource efficiency. It's not an issue when containers are small, but as images accumulate and their total size eventually reaches 40-50GB, the wasted resources can become quite significant, which is why such optimization work is important. [^2][^3][^4]

## Contribution

After working through all the challenges, I finally succeeded in merging the Apache-Spark kernel image I created.

 ![](https://cdn.lablup.com/kangmin_intern_review_img10_0a1c0f1197.png) 


This internship was a very meaningful experience where I got to work with the **backend.ai** product and contribute by creating kernel images that people can use directly. Due to the impact of COVID-19, most of the work was remote, although I felt I would have enjoyed it more if I could have worked at the company office. On the last day, Jeonggyu mentioned that **it was unfortunate we couldn't have a more in-depth internship where interns could ask questions in person about things they didn't know, and we could answer them directly**, which I completely agreed with.

Working from home made it significantly **harder to stay focused** compared to working at the office. It felt about 1.5 times more difficult.
Next time, I hope to have the opportunity to meet face-to-face, chat about various things, and hear diverse perspectives beyond just development.
**It was a valuable experience working with excellent people at a great company.**

[^1]:
    [https://hoyy.github.io/posts/spark-start-install-standalone](https://hoyy.github.io/posts/spark-start-install-standalone)

[^2]:
    [https://devaom.tistory.com/5](https://devaom.tistory.com/5)

[^3]:
    [https://www.joinc.co.kr/w/man/12/docker/storage](https://www.joinc.co.kr/w/man/12/docker/storage)

[^4]:
    [https://blog.naver.com/alice\_k106/221530340759](https://blog.naver.com/alice_k106/221530340759)

A 2020 winter internship at Lablup, building kernel images like Apache Spark for Backend.AI and working through the problems that came up along the way.

2020 Winter Internship at Lablup

A glimpse into growth and learning through an intern’s eyes.