Installing (python3) airflow does not create airflow directory

Question

Having problem where installing airflow (in python3) does not create the airflow directory (unlike in the python2 version, where the directory is created automatically after pip install).

[airflow@airflowetl ~]$ pip3 --no-cache-dir install apache-airflow --user
Collecting apache-airflow
  Downloading https://files.pythonhosted.org/packages/d5/12/6e5e4207483df7e82ac87079281f8f500e6610247ceb1464ed5592f4ceed/apache_airflow-1.10.5-py2.py3-none-any.whl (5.9MB)
    78% |█████████████████████████▎      | 4.6MB 1.4MB/s eta 0:00:01debug2: channel 0: window 999183 sent adjust 49393
    100% |████████████████████████████████| 5.9MB 1.4MB/s
.....
Installing collected packages: apache-airflow
Successfully installed apache-airflow-1.10.5



[airflow@airflowetl ~]$ pwd
/home/airflow
[airflow@airflowetl ~]$ ls
Documents  Downloads  projects  tmp

Any idea what is happening here and how to fix?

Answer 1

Found that unlike with python2, airflow in python3 does not create the ~/airflow dir until the first time airflow is run

The first time you run Airflow, it will create a file called airflow.cfg in your $AIRFLOW_HOME directory (~/airflow by default). https://airflow.readthedocs.io/en/1.9.0/configuration.html#setting-configuration-options

I am assuming this means any airflow ... CLI command, but in my case I just ran the next installation step (though I assume something like airflow list_dags would also work)...

[airflow@airflowetl ~]$ airflow initdb
[2019-10-18 13:36:56,916] {__init__.py:51} INFO - Using executor SequentialExecutor
DB: sqlite:////home/airflow/airflow/airflow.db
[2019-10-18 13:36:57,438] {db.py:369} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current schema
INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
/home/airflow/.local/lib/python3.6/site-packages/alembic/ddl/sqlite.py:39: UserWarning: Skipping unsupported ALTER for creation of implicit constraint
  "Skipping unsupported ALTER for "
....
INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 004c1210f153, increase queue name size limit
WARNI [airflow.utils.log.logging_mixin.LoggingMixin] cryptography not found - values will not be stored encrypted.
Done.
[airflow@airflowetl ~]$ cd
[airflow@airflowetl ~]$ ls
airflow
[airflow@airflowetl ~]$ tree airflow/
airflow/
├── airflow.cfg
├── airflow.db
├── logs
│   └── scheduler
│       ├── 2019-10-18
│       └── latest -> /home/airflow/airflow/logs/scheduler/2019-10-18
└── unittests.cfg

4 directories, 3 files
[airflow@airflowetl ~]$ mkdir airflow/dags

---

* (For others having airflow install problems) Note also that the airflow CLI binary location was not initially in my path; had to do a quick

[airflow@airflowetl ~]$ cd
[airflow@airflowetl ~]$ pwd
/home/airflow
[airflow@airflowetl ~]$ find . -name airflow
./.local/lib/python3.6/site-packages/airflow
./.local/lib/python3.6/site-packages/airflow/bin/airflow
./.local/lib/python3.6/site-packages/airflow/www/templates/airflow
./.local/lib/python3.6/site-packages/airflow/www_rbac/templates/airflow
./.local/bin/airflow
[airflow@airflowetl ~]$ echo "PATH=$PATH:~/.local/bin" >> ~/.bashrc
[airflow@airflowetl ~]$ source ~/.bashrc

before could use the binary (another problem I did not have when installing airflow for python2).