FreeBSD Getting Started with DEVPI Server - Private PyPI Python Packages

DevPI is a server written in Python, making it highly compatible with PyPI. Because DevPI is written in Python, you can run DevPI locally or can be installed on a private server such as FreeBSD. Simplicity and ease of use are one of the advantages of DevPI, so it is natural that the DevPI installation process is not too difficult.

DevPi was released with three Python packages, namely:

1. devpi-server
As a caching proxy to PyPI. DevPI server is capable of serving consistent caching indexes to pypi.python.org as well as local github-style overlay indexes. Once you have the pypi DevPI package, it is unchanged and will never be changed.

2. devpi-web
One of the devpi-server plugins that provides a web interface and search. With DevPI Web you can search local package directories easily.
 
3. devpi-client
One of DevPI's command line features with sub commands for creating users, using indexes, uploading and installing from indexes, as well as a “test” command to run tox. The DevPI client is indispensable in more esoteric use cases.




In this tutorial we will create a mirror with PyPI, one of the Python platforms for fetching and downloading python packages. Pypi mirror is very useful if you want to have a personal or internal use copy of PyPI. Website mirroring refers to the concept of replicating or copying a website or any network node with a purpose
  1. Ensure the availability of the original site.
  2. Provides real-time backup of the original site.
  3. Reduces network traffic.
  4. increase access speed.
To do the job, we will use a package called DevPi. Installing and running DevPI is very easy, in FreeBSD you have to run the Python virtual environment.


1. Create Python Venv

As we know, FreeBSD has a lot of Python packages in its repository. To avoid version differences and so on, FreeBSD users are required to install Python packages in a virtual environment. This aims to isolate each installed package so that it is not connected to other Python packages. Below we will explain how to install a Python virtual environment (venv) on FreeBSD.

Create directory Python venv
root@ns3:~ # mkdir -p /var/venv
root@ns3:~ # cd /var/venv
root@ns3:/var/venv # python -m venv devpi
After that, you activate the Python virtual environment.

Activate Python venv
root@ns3:/var/venv # cd devpi
root@ns3:/var/venv/devpi # source bin/activate.csh
(devpi) root@ns3:/var/venv/devpi #
By running the command above, it means you have activated Python venv. Then you run the upgrade pip command.

Activate Python venv
(devpi) root@ns3:/var/venv/devpi # pip install --upgrade pip


2. Install DevPI Server

In this section we will begin installing and running the DevPI server to serve an automatically and efficiently updated PyPI cache mirror on your FreeBSD server. Devpi server uses SQLite to store its data, on FreeBSD you don't need to install SQLite because it is automatically installed by FreeBSD. OK, let's start installing devpi-server, devpi-client and devpi-web.

Install DevPI
(devpi) root@ns3:/var/venv/devpi # pip install -q -U devpi-server devpi-web devpi-client
After that, check the DevPI version of the server you are using with the following command.

Check version
(devpi) root@ns3:/var/venv/devpi # devpi-server --version
6.10.0
The next step is to initialize the devpi server.

Initialize Devpi server
(devpi) root@ns3:/var/venv/devpi # devpi-init --serverdir /root/.devpi/server


3. Create pip.conf

On FreeBSD the pip configuration file for the DevPI server is located in the Python virtual environment, namely "/var/venv/devpi". On each different operating system, the location of the pip.conf file is different. The pip.conf file can be used to define the configuration and environment variables that pip runs, such as:
  1. command line options.
  2. environmental variables.
  3. configuration file
pip has 3 "levels" of configuration files:
  1. global: system-wide configuration file, shared across users.
  2. user: configuration file per user.
  3. site: configuration files per environment; namely per-virtualenv.
Additionally, on the DevPI server the pip.conf file can be used to avoid retyping the index URL with pip or easy-install. You can feel the benefits of the pip.conf file when installing applications with the pip command.

Below we provide an example of a pip.conf file configuration script.

Create pip.conf
(devpi) root@ns3:/var/venv/devpi # touch pip.conf
(devpi) root@ns3:/var/venv/devpi # ee pip.conf
[global]
index-url = http://192.168.5.2:3141/root/pypi/+simple/

[search]
index = http://192.168.5.2:3141/root/pypi/

[install]
Trusted-host = 192.168.5.2
After that create a .pydistutils.cfg file to configure easy_install. Add the script below to the .pydistutils.cfg file.

Create .pydistutils.cfg
(devpi) root@ns3:/var/venv/devpi # touch .pydistutils.cfg
(devpi) root@ns3:/var/venv/devpi # ee .pydistutils.cfg
[easy_install]
index_url = http://192.168.5.2:3141/root/pypi/+simple/


4. Enable Devpi Server with Python Supervisor

Because the DevPI server is written in Python and runs in a Python virtual environment, to enable the DevPI server daemon to run in the FreeBSD background we use the Python supervisor. If you are still in the Python virtual environment, exit Python Venv, then install python supervisor.

Exit Python Venv and Install py39-supervisor
(devpi) root@ns3:/var/venv/devpi # deactivate
root@ns3:/var/venv/devpi # pkg install py39-supervisor
After that, create an rc.d script so that Python supervisor can run automatically. In the /ec/rc.conf file, add the script below.

Script rc.conf (enable py39-supervisor)
root@ns3:/var/venv/devpi # ee /etc/rc.conf
supervisord_enable="YES"
supervisord_config="/usr/local/etc/supervisord.conf"
supervisord_user="root"
Then, open the supervisord.conf file, delete all the scripts and replace them with the scripts below.

/usr/local/etc
root@ns3:/var/venv/devpi # cd /usr/local/etc
root@ns3:/usr/local/etc # ee supervisord.conf
[unix_http_server]
file=/var/run/supervisor/supervisor.sock   ; the path to the socket file
;chmod=0700                 ; socket file mode (default 0700)
;chown=nobody:nogroup       ; socket file uid:gid owner
;username=user              ; default is no username (open server)
;password=123               ; default is no password (open server)

;[inet_http_server]         ; inet (TCP) server disabled by default
;port=127.0.0.1:9001        ; ip_address:port specifier, *:port for all iface
;username=user              ; default is no username (open server)
;password=123               ; default is no password (open server)

[supervisord]
logfile=/var/log/supervisord.log ; main log file; default $CWD/supervisord.log
logfile_maxbytes=50MB        ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=10           ; # of main logfile backups; 0 means none, default 10
loglevel=info                ; log level; default info; others: debug,warn,trace
pidfile=/var/run/supervisor/supervisord.pid ; supervisord pidfile; default supervisord.pid
nodaemon=false               ; start in foreground if true; default false
silent=false                 ; no logs to stdout if true; default false
minfds=1024                  ; min. avail startup file descriptors; default 1024
minprocs=200                 ; min. avail process descriptors;default 200
;umask=022                   ; process file creation umask; default 022
;user=supervisord            ; setuid to this UNIX account at startup; recommended if root
;identifier=supervisor       ; supervisord identifier, default is 'supervisor'
;directory=/tmp              ; default is not to cd during start
;nocleanup=true              ; don't clean up tempfiles at start; default false
;childlogdir=/tmp            ; 'AUTO' child log dir, default $TEMP
;environment=KEY="value"     ; key value pairs to add to environment
;strip_ansi=false            ; strip ansi escape codes in logs; def. false

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor/supervisor.sock ; use a unix:// URL  for a unix socket
;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
;username=chris              ; should be same as in [*_http_server] if set
;password=123                ; should be same as in [*_http_server] if set
;prompt=mysupervisor         ; cmd line prompt (default "supervisor")
;history_file=~/.sc_history  ; use readline history if available

[program:devpi]
command=/usr/local/bin/devpi-server  --port 3141 --host 192.168.5.2 --serverdir /root/.devpi/server              ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1                    ; number of processes copies to start (def 1)
directory=/root/.devpi                ; directory to cwd to before exec (def no cwd)
;umask=022                     ; umask for process (default None)
;priority=999                  ; the relative start priority (default 999)
autostart=true                ; start at supervisord start (default: true)
;startsecs=1                   ; # of secs prog must stay up to be running (def. 1)
startretries=3                ; max # of serial start failures when starting (default 3)
autorestart=unexpected        ; when to restart if exited after running (def: unexpected)
;exitcodes=0                   ; 'expected' exit codes used with autorestart (default 0)
;stopsignal=QUIT               ; signal used to kill process (default TERM)
;stopwaitsecs=10               ; max num secs to wait b4 SIGKILL (default 10)
stopasgroup=true             ; send stop signal to the UNIX process group (default false)
killasgroup=true             ; SIGKILL the UNIX process group (def false)
user=root                   ; setuid to this UNIX account to run the program
redirect_stderr=true          ; redirect proc stderr to stdout (default false)
;stdout_logfile=/var/venv/devpi/supervisord.log        ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10     ; # of stdout logfile backups (0 means none, default 10)
;stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
;stdout_events_enabled=false   ; emit events on stdout writes (default false)
;stdout_syslog=false           ; send stdout to syslog with process name (default false)
;stderr_logfile=/a/path        ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups=10     ; # of stderr logfile backups (0 means none, default 10)
;stderr_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
;stderr_events_enabled=false   ; emit events on stderr writes (default false)
;stderr_syslog=false           ; send stderr to syslog with process name (default false)
;environment=A="1",B="2"       ; process environment additions (def no adds)
;serverurl=AUTO                ; override serverurl computation (childutils)

;[eventlistener:theeventlistenername]
;command=/bin/eventlistener    ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1                    ; number of processes copies to start (def 1)
;events=EVENT                  ; event notif. types to subscribe to (req'd)
;buffer_size=10                ; event buffer queue size (default 10)
;directory=/tmp                ; directory to cwd to before exec (def no cwd)
;umask=022                     ; umask for process (default None)
;priority=-1                   ; the relative start priority (default -1)
;autostart=true                ; start at supervisord start (default: true)
;startsecs=1                   ; # of secs prog must stay up to be running (def. 1)
;startretries=3                ; max # of serial start failures when starting (default 3)
;autorestart=unexpected        ; autorestart if exited after running (def: unexpected)
;exitcodes=0                   ; 'expected' exit codes used with autorestart (default 0)
;stopsignal=QUIT               ; signal used to kill process (default TERM)
;stopwaitsecs=10               ; max num secs to wait b4 SIGKILL (default 10)
;stopasgroup=false             ; send stop signal to the UNIX process group (default false)
;killasgroup=false             ; SIGKILL the UNIX process group (def false)
;user=chrism                   ; setuid to this UNIX account to run the program
;redirect_stderr=false         ; redirect_stderr=true is not allowed for eventlisteners
;stdout_logfile=/a/path        ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10     ; # of stdout logfile backups (0 means none, default 10)
;stdout_events_enabled=false   ; emit events on stdout writes (default false)
;stdout_syslog=false           ; send stdout to syslog with process name (default false)
;stderr_logfile=/a/path        ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups=10     ; # of stderr logfile backups (0 means none, default 10)
;stderr_events_enabled=false   ; emit events on stderr writes (default false)
;stderr_syslog=false           ; send stderr to syslog with process name (default false)
;environment=A="1",B="2"       ; process environment additions
;serverurl=AUTO                ; override serverurl computation (childutils)

;[group:thegroupname]
;programs=progname1,progname2  ; each refers to 'x' in [program:x] definitions
;priority=999                  ; the relative start priority (default 999)

; The [include] section can just contain the "files" setting.  This
; setting can list multiple files (separated by whitespace or
; newlines).  It can also contain wildcards.  The filenames are
; interpreted as relative to this file.  Included files *cannot*
; include files themselves.

;[include]
;files = /var/venv/devpi/gen-config/supervisor-devpi.conf
Enable Python supervisor by running the command below.

Restart Python supervisor
root@ns3:/usr/local/etc # service supervisord restart
Stopping supervisord.
Waiting for PIDS: 965.
Starting supervisord.


5. User management DevPI Client

After you activate the DevPI server, the next step is to create a user and password to connect to the DevPI server. This step is very important because it maintains your DevPI server network. Follow the guide below to create a DevPI client user and password.

Enbale Python Venv and Connect the DevPI client to the DevPI server
root@ns3:~ # cd /usr/local/etc
root@ns3:/usr/local/etc # cd /var/venv/devpi
root@ns3:/var/venv/devpi # source bin/activate.csh
(devpi) root@ns3:/var/venv/devpi # devpi use -l http://192.168.5.2:3141
The command above is used to activate Python Venv and the command "devpi use -l" to connect the client to the server. After the client connects to the server, create a json file, see the example below.

Create "/root/.devpi/client/current.json" file
(devpi) root@ns3:/var/venv/devpi # touch /root/.devpi/client/current.json
(devpi) root@ns3:/var/venv/devpi # ee /root/.devpi/client/current.json
{
  "features": [
    "server-keyvalue-parsing"
  ],
  "login": "http://192.168.5.2:3141/+login"
}

a. Change the root password

The root password is generated automatically when DevPI server is installed. For security reasons, change the root password.

Change the root password
(devpi) root@ns3:/var/venv/devpi # devpi login root --password=''
(devpi) root@ns3:/var/venv/devpi # devpi user -m root password=router1234
The command devpi login root --password='' is used when you first install the DevPI server. Because the root user password after you finish installing DevPI server, the default is blank: ''

After you change the root password, you must log in to root with the password you created, namely "router1234". Now run the login command to root.

Login to root
(devpi) root@ns3:/var/venv/devpi # devpi login root
password for user root at http://192.168.5.2:3141/: router1234
logged in 'root', credentials valid for 10.00 hours

b. Create User

To limit DevPI server users, we will create users who can access the DevPI server.

Create new user
(devpi) root@ns3:/var/venv/devpi # devpi user -c john email=datainchi@gmail.com password=johndoe123
user created: john
The above command is used to create,
User: john
Password: johndoe123

Try now to log in with user john.

Login with user john
(devpi) root@ns3:/var/venv/devpi # devpi login john
password for user john at http://192.168.5.2:3141/: johndoe123
logged in 'john', credentials valid for 10.00 hours


6. Using pip on DevPI Server

In this section we will use the pip command on the DevPI server. Both easy_install and have the -i option to specify the index server url. We will demonstrate the use of the pip command which we direct to the installer on a dedicated root/pypi index, which is served by the devpi server by default. OK, now let's install the simplejson package as our test cache. We will log in with the root user.

Install simplejson package
(devpi) root@ns3:/var/venv/devpi # devpi login root
(devpi) root@ns3:/var/venv/devpi # pip install -i http://192.168.5.2:3141/root/pypi/+simple/ simplejson
Looking in indexes: http://192.168.5.2:3141/root/pypi/+simple/
Collecting simplejson
  Downloading http://192.168.5.2:3141/root/pypi/%2Bf/bce/df4cae0d47839/simplejson-3.19.2-py3-none-any.whl (56 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.0/57.0 kB 1.2 MB/s eta 0:00:00
We will give an example again, in the example below we log in with user john, and we will install the panda application.

Install panda package with user john
(devpi) root@ns3:/var/venv/devpi # devpi login john
(devpi) root@ns3:/var/venv/devpi # pip install --trusted-host 192.168.5.2 -i http://192.168.5.2:3141/root/pypi panda
Looking in indexes: http://192.168.5.2:3141/root/pypi
Collecting panda
  Downloading http://192.168.5.2:3141/root/pypi/%2Bf/f21/3b848f09268b3/panda-0.3.1.tar.gz (5.8 kB)
Devpi Server is a powerful and flexible Python package management system, which is used to install software. With DevPI Server you can simplify the distribution and management of existing software in Python. Although there are still weaknesses and shortcomings, the Devpi server remains a reliable and efficient tool for managing, distributing, and deploying Python software packages. We hope this article provides you with in-depth and comprehensive information on how to install Devpi server on FreeBSD.
Iwan Setiawan

I Like Adventure: Mahameru Mount, Rinjani Mount I Like Writer FreeBSD

Post a Comment

Previous Post Next Post