This page was generated from docs/notebooks/first_steps.ipynb . Interactive online version: . Download notebook .

First steps

ConFlowGen has been developed with Jupyter Notebooks in mind. While it is also possible to write your code as pure Python Scripts (see e.g. the Python Script examples), Jupyter Notebooks allow to mix the code with explanatory texts, LaTeX formulas, etc. in a single document. In addition, the generated visualisations neatly blend in.

For newcomers it is suggested to download the Anaconda distribution and first to familiarize yourself with the newly installed tools. Then, it is suggested to create a separate environment as described for the Jupyter Notebook examples. During the environment set up, ConFlowGen is automatically installed. Then, you can start JupyterLab from Anaconda Navigator or from the CLI. Please ensure that you are in the correct conda environment (it should be conflowgen) when you work.

Prerequisites

To start, we first import ConFlowGen as a module.

import os
import datetime

import pandas as pd

import conflowgen

Then, we set up the logger

logger = conflowgen.setup_logger(
    logging_directory="./data/logger",  # use subdirectory relative to Jupyter Notebook
    format_string="%(message)s"  # only show log messages, discard timestamp etc.
)

Creating log directory at ./data/logger
Creating log file at ./data/logger/2025-02-16--20-56-18.log

Database selection

Now, we select a database to work in.

database_chooser = conflowgen.DatabaseChooser(
    sqlite_databases_directory="./data/db"  # use subdirectory relative to Jupyter Notebook
)
demo_file_name = "my_demo.sqlite"

database_chooser.create_new_sqlite_database(demo_file_name, overwrite=True)

Creating SQLite directory at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/latest/docs/notebooks/data/db
No previous database detected, creating new at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/latest/docs/notebooks/data/db/my_demo.sqlite
Opening file /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/latest/docs/notebooks/data/db/my_demo.sqlite
journal_mode: wal
cache_size: -32768
page_size: 4096
foreign_keys: 1
Creating new database at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/latest/docs/notebooks/data/db/my_demo.sqlite
Creating all tables...
Seed with default values...
Number entries in table 'DeepSeaVessel': 0
Number entries in table 'Feeder': 0
Number entries in table 'Barge': 0
Number entries in table 'Train': 0
Number entries in table 'Truck': 0
Number entries in table 'Container': 0
Randomly set seed 1739739378707714208 for 'SqliteDatabaseConnection'

General settings

We use ContainerFlowGenerationManager.set_properties() to set a name, the start_date and the end_date for the generation process. The start_date and end_date parameters must be provided as a datetime.date. In our example, the end_time is set as a 21 day interval on top of the start_date which is set to the time of execution. Other general settings can be set here, see the ContainerFlowGenerationManager class for more information.

today = datetime.datetime.now().date()
container_flow_generation_manager = conflowgen.ContainerFlowGenerationManager()
container_flow_generation_manager.set_properties(
    name="Demo file",
    start_date=today,
    end_date=today + datetime.timedelta(days=21)
)

Randomly set seed 1739739378714455521 for 'TruckForImportContainersManager'
Randomly set seed 1739739378716807061 for 'TruckForExportContainersManager'
Randomly set seed 1739739378718492078 for 'ContainerFactory'
Randomly set seed 1739739378720161663 for 'LargeScheduledVehicleForOnwardTransportationManager'
Randomly set seed 1739739378728572200 for 'AllocateSpaceForContainersDeliveredByTruckService'
Replace seed 1739739378718492078 with 1739739378730071975 for 'ContainerFactory' for the new round.
Randomly set seed 1739739378731782316 for 'AssignDestinationToContainerService'
Loading destination distribution...

Creating schedules

In the next step, we add a feeder liner service to the database. We do this by using PortCallManager.add_vehicle(). The concrete vehicle instances are only generated when ContainerFlowGenerationManager.generate() is invoked.

port_call_manager = conflowgen.PortCallManager()

At first we define a name for our new feeder liner service.

feeder_service_name = "LX050"

By using PortCallManager.add_service_that_calls_terminal(), we can define the attributes for our feeder service.

• vehicle_type defines, that we deal with a feeder as the mode of transport. Other valid modes of transport are deep_sea_vessel, barge, or train.

• service_name defines a fictional name for that service.

• vehicle_arrives_at specifies the date where the first port call of this particular line is usually happening. This parameter must be a datetime.date.

• vehicle_arrives_at_time sets the scheduled timeslot of the port call. This parameter must be a datetime.time.

• average_vehicle_capacity defines the average capacity of the vessels utilized on this line. Parameter must be int or float.

• average_inbound_container_volume sets the capacity which is in average moved between the feeder and the terminal at each call. Parameter must be int or float.

• next_destinations can be set, consisting of name and frequency which can, e.g., be used as implication for storage and stacking problems. A list of name-frequency pairs is expected here.

port_call_manager.add_service_that_calls_terminal(
    vehicle_type=conflowgen.ModeOfTransport.feeder,
    service_name=feeder_service_name,
    vehicle_arrives_at=datetime.date(2021, 7, 9),
    vehicle_arrives_at_time=datetime.time(11),
    average_vehicle_capacity=800,
    average_inbound_container_volume=100,
    next_destinations=[
        ("DEBRV", 0.4),  # 40% of the containers go here...
        ("RULED", 0.6)   # and the other 60% of the containers go here.
    ]
)

Adding destination 'DEBRV' for schedule 'LX050' at the position 1 with a frequency of 40.00%
Adding destination 'RULED' for schedule 'LX050' at the position 2 with a frequency of 60.00%

Following the same principle and structure we can also add schedules for trains and deep sea vessels:

port_call_manager.add_service_that_calls_terminal(
    vehicle_type=conflowgen.ModeOfTransport.train,
    service_name="JR03A",
    vehicle_arrives_at=datetime.date(2021, 7, 12),
    vehicle_arrives_at_time=datetime.time(17),
    average_vehicle_capacity=90,
    average_inbound_container_volume=90,
    next_destinations=None  # Here we don't have containers that need to be grouped by destination
)

port_call_manager.add_service_that_calls_terminal(
    vehicle_type=conflowgen.ModeOfTransport.deep_sea_vessel,
    service_name="LX050",
    vehicle_arrives_at=datetime.date(2021, 7, 10),
    vehicle_arrives_at_time=datetime.time(19),
    average_vehicle_capacity=16000,
    average_inbound_container_volume=150,  # to speed up the code
    next_destinations=[
        ("ZADUR", 0.3),  # 30% of the containers go to ZADUR...
        ("CNSHG", 0.7)   # and the other 70% of the containers go to CNSHG.
    ]
)

Adding destination 'ZADUR' for schedule 'LX050' at the position 1 with a frequency of 30.00%
Adding destination 'CNSHG' for schedule 'LX050' at the position 2 with a frequency of 70.00%

Generate the data

After we have set some exemplary schedules, we can now come to the actual generation. The generation process of the synthetic container flow data is started with ContainerFlowGenerationManager.generate(), based on the information you set earlier. With the optional parameter overwrite you determine the behavior if already some data exists - with the default behavior overwrite=True, any previous container flow data is overwritten, with overwrite=False the generation step is skipped if any data is detected.

container_flow_generation_manager.generate()

Resetting preview and analysis cache...
Remove previous data...
Reloading properties and distributions...
Using transportation buffer of 0.2 when choosing the departing vehicles that adhere a schedule.
Use transport buffer of 0.2 for allocating containers delivered by trucks.
For vehicle type truck and storage requirement empty, the container dwell times need to range from 0h to 1353h
For vehicle type truck and storage requirement standard, the container dwell times need to range from 0h to 511h
For vehicle type truck and storage requirement reefer, the container dwell times need to range from 0h to 511h
For vehicle type truck and storage requirement dangerous_goods, the container dwell times need to range from 0h to 511h
For vehicle type train and storage requirement empty, the container dwell times need to range from 0h to 684h
For vehicle type train and storage requirement standard, the container dwell times need to range from 0h to 208h
For vehicle type train and storage requirement reefer, the container dwell times need to range from 0h to 208h
For vehicle type train and storage requirement dangerous_goods, the container dwell times need to range from 0h to 208h
For vehicle type feeder and storage requirement empty, the container dwell times need to range from 3h to 979h
For vehicle type feeder and storage requirement standard, the container dwell times need to range from 3h to 223h
For vehicle type feeder and storage requirement reefer, the container dwell times need to range from 3h to 223h
For vehicle type feeder and storage requirement dangerous_goods, the container dwell times need to range from 3h to 223h
For vehicle type deep_sea_vessel and storage requirement empty, the container dwell times need to range from 3h to 878h
For vehicle type deep_sea_vessel and storage requirement standard, the container dwell times need to range from 3h to 216h
For vehicle type deep_sea_vessel and storage requirement reefer, the container dwell times need to range from 3h to 216h
For vehicle type deep_sea_vessel and storage requirement dangerous_goods, the container dwell times need to range from 3h to 216h
For vehicle type barge and storage requirement empty, the container dwell times need to range from 0h to 691h
For vehicle type barge and storage requirement standard, the container dwell times need to range from 0h to 576h
For vehicle type barge and storage requirement reefer, the container dwell times need to range from 0h to 576h
For vehicle type barge and storage requirement dangerous_goods, the container dwell times need to range from 0h to 576h
For vehicle type truck and storage requirement empty, the container dwell times need to range from 0h to 1353h
For vehicle type truck and storage requirement standard, the container dwell times need to range from 0h to 511h
For vehicle type truck and storage requirement reefer, the container dwell times need to range from 0h to 511h
For vehicle type truck and storage requirement dangerous_goods, the container dwell times need to range from 0h to 511h
For vehicle type train and storage requirement empty, the container dwell times need to range from 0h to 828h
For vehicle type train and storage requirement standard, the container dwell times need to range from 0h to 122h
For vehicle type train and storage requirement reefer, the container dwell times need to range from 0h to 122h
For vehicle type train and storage requirement dangerous_goods, the container dwell times need to range from 0h to 122h
For vehicle type feeder and storage requirement empty, the container dwell times need to range from 12h to 936h
For vehicle type feeder and storage requirement standard, the container dwell times need to range from 12h to 257h
For vehicle type feeder and storage requirement reefer, the container dwell times need to range from 12h to 257h
For vehicle type feeder and storage requirement dangerous_goods, the container dwell times need to range from 12h to 257h
For vehicle type deep_sea_vessel and storage requirement empty, the container dwell times need to range from 12h to 964h
For vehicle type deep_sea_vessel and storage requirement standard, the container dwell times need to range from 12h to 468h
For vehicle type deep_sea_vessel and storage requirement reefer, the container dwell times need to range from 12h to 468h
For vehicle type deep_sea_vessel and storage requirement dangerous_goods, the container dwell times need to range from 12h to 468h
For vehicle type barge and storage requirement empty, the container dwell times need to range from 0h to 532h
For vehicle type barge and storage requirement standard, the container dwell times need to range from 0h to 720h
For vehicle type barge and storage requirement reefer, the container dwell times need to range from 0h to 720h
For vehicle type barge and storage requirement dangerous_goods, the container dwell times need to range from 0h to 720h
Loading destination distribution...
Load destination distribution for service 'LX050' by feeder with 2 destinations
Load destination distribution for service 'LX050' by deep_sea_vessel with 2 destinations
Create fleet including their delivered containers for given time range for each schedule...
Create vehicles and containers for service 'JR03A' of type 'train', progress: 1 / 3 (33.33%)
Create vehicles and containers for service 'LX050' of type 'feeder', progress: 2 / 3 (66.67%)
Create vehicles and containers for service 'LX050' of type 'deep_sea_vessel', progress: 3 / 3 (100.00%)
Loading status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.78
Stddev:  0.55
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 90.00
Maximum: 180.00
Mean:    130.00
Stddev:  39.69
(rounding errors might exist)

Assign containers that are picked up from the terminal by a vehicle adhering a schedule to their specific vehicle instance...
Assign containers to departing vehicles that move according to a schedule...
In total, 491 containers continue their journey on a vehicle that adhere to a schedule, assigning these containers to their respective vehicles...
Progress: 1 / 491 (0.20%) containers have been assigned to a scheduled vehicle to leave the terminal again.
Progress: 491 / 491 (100.00%) containers have been assigned to a scheduled vehicle to leave the terminal again.
Containers for which no outgoing vehicle adhering to a schedule could be found: 83.30%. These will be re-assigned to another vehicle type, such as a truck.
All containers for which a departing vehicle that moves according to a schedule was available have been assigned to one.
Loading status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.78
Stddev:  0.55
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 0.75
Maximum: 129.50
Mean:    55.19
Stddev:  42.76
(rounding errors might exist)

Generate trucks that pick up containers...
In total 199 containers are picked up by truck, creating these trucks now...
Progress: 1 / 199 (0.50%) trucks generated to pick up containers at the terminal.
Progress: 199 / 199 (100.00%) trucks generated to pick up containers at the terminal.
All 199 trucks that pick up a container have been generated, moving 330.75 TEU.
Generate containers that are delivered by trucks...
Progress: 1 / 129 (0.78%) of the containers which are delivered by truck are allocated on a vehicle adhering to a schedule
Vehicle '3' of type 'deep_sea_vessel' has no remaining capacity. The free capacity of 0.75 TEU is less than the required 2.5 TEU.
Progress: 129 / 129 (100.00%) of the containers which are delivered by truck are allocated on a vehicle adhering to a schedule
All 129 containers that need to be delivered by truck have been assigned to a vehicle that adheres to a schedule, corresponding to 200.5 TEU.
Loading status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.78
Stddev:  0.55
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 0.75
Maximum: 90.00
Mean:    32.92
Stddev:  29.98
(rounding errors might exist)

Generate trucks that deliver containers...
In total 129 containers are delivered by truck, creating these trucks now...
Progress: 1 / 129 (0.78%) trucks generated to deliver containers to the terminal.
Progress: 129 / 129 (100.00%) trucks generated to deliver containers to the terminal.
All 129 trucks that deliver a container are created now, moving 200.5 TEU.
Assign containers to next destinations...
Assign destinations to containers that leave the terminal with the service 'LX050' of the vehicle type feeder, progress: 1 / 2 (50.00%)
Assign destinations to containers that leave the terminal with the service 'LX050' of the vehicle type deep_sea_vessel, progress: 2 / 2 (100.00%)
Container flow generation finished
Final capacity status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.78
Stddev:  0.55
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 0.75
Maximum: 90.00
Mean:    32.92
Stddev:  29.98
(rounding errors might exist)

Export the data

The data is exported with two lines of code.

export_container_flow_manager = conflowgen.ExportContainerFlowManager()
path_to_exported_data = export_container_flow_manager.export("first_steps", "./data/export", overwrite=True)

Creating export folder ./data/export
Creating folder at ./data/export/first_steps
Converting SQL database into file format '.csv'
Resolving column destination of model <Model: Container>...
This is a nested call to resolve the column 'destination'
Gathering data for generating the 'deep_sea_vessels' table...
Resolving column large_scheduled_vehicle of model <Model: DeepSeaVessel>...
This is a nested call to resolve the column 'large_scheduled_vehicle'
Gathering data for generating the 'feeders' table...
Resolving column large_scheduled_vehicle of model <Model: Feeder>...
Gathering data for generating the 'barges' table...
No content found for the barges table, the file will be empty.
Gathering data for generating the 'trains' table...
Resolving column large_scheduled_vehicle of model <Model: Train>...
Resolving column truck_arrival_information_for_pickup of model <Model: Truck>...
This is a nested call to resolve the column 'truck_arrival_information_for_pickup'
Resolving column truck_arrival_information_for_delivery of model <Model: Truck>...
This is a nested call to resolve the column 'truck_arrival_information_for_delivery'
Saving file containers.csv
Saving file deep_sea_vessels.csv
Saving file feeders.csv
Saving file barges.csv
Saving file trains.csv
Saving file trucks.csv
Export has finished successfully.

Examining the exported data

The CSV files we can open e.g. with pandas. Here, the tabular data is presented.

df_containers = pd.read_csv(
    os.path.join(path_to_exported_data, "containers.csv"),
    index_col="id",
    dtype={
        "delivered_by_truck": "Int64",
        "picked_up_by_truck": "Int64",
        "delivered_by_vehicle": "Int64",
        "picked_up_by_vehicle": "Int64",
        "destination_sequence_id": "Int64"
    }
)
df_containers

	weight	length	storage_requirement	delivered_by	picked_up_by_initial	picked_up_by	delivered_by_vehicle	delivered_by_truck	picked_up_by_vehicle	picked_up_by_truck	emergency_pickup	destination_sequence_id	destination_name
id
1	8	20	standard	train	deep_sea_vessel	deep_sea_vessel	1	<NA>	7	<NA>	False	2	CNSHG
2	4	40	empty	train	deep_sea_vessel	deep_sea_vessel	1	<NA>	8	<NA>	False	1	ZADUR
3	22	40	dangerous_goods	train	deep_sea_vessel	deep_sea_vessel	1	<NA>	7	<NA>	False	2	CNSHG
4	12	20	dangerous_goods	train	deep_sea_vessel	deep_sea_vessel	1	<NA>	7	<NA>	False	1	ZADUR
5	4	40	empty	train	feeder	feeder	1	<NA>	6	<NA>	False	2	RULED
...	...	...	...	...	...	...	...	...	...	...	...	...	...
745	10	40	standard	truck	deep_sea_vessel	deep_sea_vessel	<NA>	301	7	<NA>	False	1	ZADUR
746	2	20	standard	truck	deep_sea_vessel	deep_sea_vessel	<NA>	298	7	<NA>	False	2	CNSHG
747	16	40	standard	truck	deep_sea_vessel	deep_sea_vessel	<NA>	275	7	<NA>	False	1	ZADUR
748	8	20	standard	truck	deep_sea_vessel	deep_sea_vessel	<NA>	235	7	<NA>	False	1	ZADUR
749	28	40	standard	truck	deep_sea_vessel	deep_sea_vessel	<NA>	303	7	<NA>	False	2	CNSHG

749 rows × 13 columns

The column delivered_by contains the vehicle type a container is delivered by. For trucks, the column delivered_by_truck mentions the corresponding ID of the vehicle that is stored in the separate table trucks.csv. The same scheme applies to picked_up_by and picked_up_by_truck.

df_trucks = pd.read_csv(
    os.path.join(path_to_exported_data, "trucks.csv"),
    index_col="id"
)
df_trucks

	delivers_container	picks_up_container	realized_container_pickup_time	realized_container_delivery_time
id
1	False	True	2025-02-24 16:12:54.342410	NaN
2	False	True	2025-03-03 11:38:06.823061	NaN
3	False	True	2025-03-04 13:42:56.110884	NaN
4	False	True	2025-02-27 15:17:50.133445	NaN
5	False	True	2025-03-11 13:35:53.424699	NaN
...	...	...	...	...
324	True	False	NaN	2025-02-11 21:38:28.716789
325	True	False	NaN	2025-02-18 06:14:42.396287
326	True	False	NaN	2025-03-05 11:46:01.158810
327	True	False	NaN	2025-02-04 15:27:30.969015
328	True	False	NaN	2025-02-24 13:02:06.024822

328 rows × 4 columns

Here, we can see which trucks deliver a container, pick up a container, and for which time their arrivals are realized.

All vehicle types except trucks are considered vehicles that typically arrive according to a schedule and which move larger amounts of containers at once. Thus, the table has a slightly different shape. They all have ids from the same ID range and for each vehicle type the ids might be non-consecutive. So e.g. for trains, in the column delivered_by_vehicle in the container table the ID is mentioned that we can look up in the table of trains.csv.

df_trains = pd.read_csv(
    os.path.join(path_to_exported_data, "trains.csv"),
    index_col="id"
)
df_trains

	vehicle_name	capacity_in_teu	inbound_container_volume	realized_arrival
id
1	1	90	90	2025-02-17 17:00:00
2	2	90	90	2025-02-24 17:00:00
3	3	90	90	2025-03-03 17:00:00

Corresponding CSV files exist for the other vehicles as well.