How to work properly

Docker

What is a container?

A container is process on your machine that is isolated from all other processes. This isolation stands on kernels namespaces which is a feature that has been in Linux for a long time. Docker makes it easy to use. A container is:

  • A runnable instance of an image
  • Can be run on local machines, virtual machines or deployed on the cloud
  • Is portable (can be run on any OS)
  • Is isolated from other containers and runs its own software, binaries, and configurations

What is container’s image?

When running a container, it uses an isolated file system. This custome filesystem is provided by a container image (contains everything to run an application : dependencies, configurations, scripts, binaries, environment variables, default command to run, metadata)

How to containerize an application?

We need to build the container image of an application using Dockerfile. A Docker file is simply a text-based file with no file extension. A Dockerfile contains a script of instruction that Docker uses to create a container image.

  1. In the app directory, create a file named DockerFile
  2. Edit the DockerFile
    1. FROM
    2. WORKDIR
    3. COPY
    4. RUN
    5. CMD
    6. EXPOSE
  3. Build the container image with the following command
  4. docker build -t getting-started .

Start an app container

Now that  we have an image, we can run the application in a container using the docker run command.

docker run -dp 3000:3000 getting-started
-d to run the container in detach mode

In Docker Desktop, go to containers tab to see a list of your containers.

One major issue that we have now, is that the data is not persistent. When our container is deleted or stopped the data disappear. In order to solve this issue, we need to create a container Volume and mount it when we launch the application.

Note : Docker Desktop is a paying version, Docker on Linux is free.

How to work properly

Use of a Virtual environment

  • Create a folder to work in
  • Create a python environment with the following command:
    MS DOS
    1
    python-m venv “environnement name”
  • Activate environment with
    MS DOS
    1
    .\nom environment\Scripts\activate

     

Use Git and GitLab or Github

  • Git init your App
  • Make your first commit
  • I would recommand to use the git extension in Visual Code Studio

Use of Pyenv

  • Pyenv allows to manager version of python

Structure of projects

App:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
/data : flat files
 
/queries : test queries
 
/scripts : batch command
 
/src
 
/utils
 
Main.py
 
/venv_graph
 
Docker-compose.yml
 
Makefile
 
Requirements.txt
 
README

Make the difference with Visual Studio

  • Explorer : folder and file structures
  • Search
  • Source Control : Git Management, make sure to display the changes in for of a tree, and get ready to make your first commit
  • Install plugins :
    • Dataiku DSS
    • Docker
    • Python
    • Pylance
    • Powershell
  • Testing: define your testing strategies
  • Powershell : Quick access to windows powershell
  • Dataiku : connect to your dss project via API key
  • Docker : manage images and containers
  • Connect git to remote location (Gitlab, Github, etc…)

Technical Stack choices

  • Databases : Neo4j (Graph), MySQL, Amazon RDS
  • Frontend : Reactjs + Vuejs (Typescript)
  • Backend : Flask API, Django (python)
  • Versioning + CI/CD : Gitlab, Github
  • Process manager : Supervisor or Circus (python)
  • Data Update : Airflow
  • Cloud server : AWS EC2, S3, Docker
  • API Key, User And Password as Environment Variables on the server side (you can use a .env file or .secrets file)
  • Authentication : Auth0
  • Payment System : Stripe (Publishable Key on the front, Secret Key on the Flask back)
    • npm install @stripe/react-stripe-js @stripe/stripe-js
    • pip install stripe
    • create payments database in MYSQL and write information in it using Flask api
  • Analytics : Google analytics

Project structure

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
my-saas-app/
│
├── backend/                   # Flask Backend
│   ├── app/                   # Your Flask application code
│   │   ├── __init__.py        # Application initialization
│   │   ├── routes.py          # Flask routes
│   │   ├── models.py          # Database models
│   │   ├── services.py        # Business logic (payment, etc.)
│   │   └── utils.py           # Helper functions
│   ├── config.py              # Configuration for Flask (DB, Stripe keys, etc.)
│   ├── requirements.txt       # Python dependencies
│   └── Dockerfile             # Dockerfile for the Flask app
│
├── frontend/                  # ReactJS Frontend
│   ├── public/                # Public static files (index.html, etc.)
│   ├── src/                   # React source code
│   │   ├── components/        # Reusable components (forms, buttons, etc.)
│   │   ├── pages/             # React pages (e.g., Home, Payment, etc.)
│   │   ├── App.js             # Main React app
│   │   └── index.js           # Entry point
│   ├── package.json           # Node.js dependencies
│   └── Dockerfile             # Dockerfile for React app
│
├── db/                        # Database related files
│   ├── neo4j/                 # Neo4j-specific configurations
│   └── mysql/                 # MySQL-specific configurations
│
├── docker-compose.yml         # Docker Compose to manage all containers
└── README.md                  # Documentation

Graph Neptune

Graph transversal : goes through a graph to retrieve interesting data regarding a specific query.

Amazon Neptune is a fully managed graph database service. The core is a graph database engine.

Neptune supports : Apache Tinkerpop Gremlins, Neo4J openCypher

  1. Create a Neptune DB Cluster
  2. Connecting to Neptune graph : Snowflake and Neptune are in 2 different VPC
  3. Use curl and ssL certificate to communicate with Neptune endpoint
  4. Use Neptune Bulk Loader to ingest large amounts of data
  5. Use Graph query langage to query data
  6. Use vizualisation tools : Graph-explorer, Tom Sawyer, Cambridge Intelligence, Graphistry, … or custom app
  7. Export data using curl
  8. Use Amazon lambda to retrieve data or query the graph on a regular basis
  9. Neptune ML setup and SageMaker : Neptune supports both transductive inference, which returns predictions that were pre-computed at the time of training, based on your graph data at that time, and inductive inference, which returns applies data processing and model evaluation in real time, based on current data. See The difference between inductive and transductive inference.

 

Neptune ML can train machine learning models to support five different categories of inference:

  • Node classification
  • Node regression
  • Edge classification
  • Edge regression
  • Link prediction

Max size : 768GB

Neo4j

Neo4j graph can be pulled using docker image. Loading data into the graph is quite easy, and only needs to follow a naming framework for nodes and relationships csv files.

To start a neo4j project from scratch we need :

docker-compose.yml

  • YAML
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    # docker version file
    version: "3.8"
    # list of services
    services:
    #name of the service
    neo4j:
      image: neo4j:5.8.0
      container_name: neo4j-graph
      ports:
        - 7474:7474
        - 7687:7687
      environment:
        - NEO4J_AUTH=none
        - NEO4J_dbms_memory_transaction_total_max=4G
        - NEO4J_dbms_connector_bolt_listen__address=:7687
        - NEO4J_dbms_connector_http_enabled=true
        - NEO4J_dbms_connector_https_enable d=true
        - NEO4J_dbms_memory_heap_max__size=4g
      volumes:
        - ./data/:/var/lib/neo4j/import/
        - ./data/:/var/lib/neo4j/data/

     

neo4j

  • YAML
    1
    2
    FROM neo4j:5.8.0
    RUN ["apt-get", "-y", "install", "vim"]

Makefile

 

YAML
1
2
3
4
5
6
7
build-images :
  docker build -f "Dockerfile.neo4j" -t "neo4j-icij" "."
docker-compose :
  make build-images
  docker image prune -f
  docker volume prune -f
  docker compose up

Import data with when you are in the container (use docker exec -it command)

  • MS DOS
    1
    2
    3
    4
    5
    6
    7
    8
    9
    bin/neo4j-admin database import full \
    --nodes="import/NODES.csv" \
    --relationships="import/RELATIONSHIPS.csv" \
    --delimiter="," \
    --array-delimiter="\t" \
    --skip-bad-relationships=true \
    --overwrite-destination \
    --multiline-fields=true \
    neo4j

     

Run a docker image

1
docker run -d -p 8080:80 nginx

Entrer dans un conteneur docker

1
docker exec -it ID_CONTENEUR /bin/bash

Stop a docker image / delete a docker image

1
2
3
docker stop ID_CONTENEUR
 
docker rm ID_CONTENEUR

Pull an existing image

1
docker pull hello-world

Display all active containers

1
2
3
4
5
docker ps
 
or
 
docker container ls

Display all existing images

1
docker images -a

System cleaning

1
2
3
4
5
6
7
8
9
docker system prune
 
sudo make docker-compose
 
sudo docker ps
 
sudo docker exec -it cma-cgm-neo4j-internal /bin/bash
 
sudo docker restart cma-cgm-neo4j-internal

  • Open folder in Visual Studio
  • Create git
  • Create virtual environment with venv
  • Activate the environment by running Activate.ps1 in Scripts folder
  • Create a requirements.txt
  • Gitignore venv
  • Start creating your files
  • Code and debug
  • Containerize?
  • Deliver to your server (CI/CD)

Flask is running at port 5000

React is running at port 3000

Neo4j is running on port 7687 and is accessible on port 7474 via the browser

Gestion de Git

  1. Git clone (ou git pull plus tard) : (git pull origin master)
  2. Git checkout -b (créer une branche)
  3. Git checkout « nom de branche » : change de branche
  4. Code sur la branche
  5. Git add. (ajouter les fichiers aux commits)
  6. Git status
  7. Git commit -m “maj function “
  8. Git push -u origin « nom de la branche » (origin est le nom du remote)
  9. Proposition de merge, « create merge request »
  10. Git pull

Pour tester, il vaut mieux récupérer une branche avant de merger.

Edit the .gitignore file

.gitignore : key, libraries, data