First steps

ConFlowGen has been developed with Jupyter Notebooks in mind. While it is also possible to write your code as pure Python Scripts (see e.g. the Python Script examples), Jupyter Notebooks allow to mix the code with explanatory texts, LaTeX formulas, etc. in a single document. In addition, the generated visualisations neatly blend in.

For newcomers it is suggested to download the Anaconda distribution and first to familiarize yourself with the newly installed tools. Then, it is suggested to create a separate environment as described for the Jupyter Notebook examples. During the environment set up, ConFlowGen is automatically installed. Then, you can start JupyterLab from Anaconda Navigator or from the CLI. Please ensure that you are in the correct conda environment (it should be conflowgen) when you work.

Prerequisites

To start, we first import ConFlowGen as a module.

import os
import datetime

import pandas as pd

import conflowgen

Then, we set up the logger

logger = conflowgen.setup_logger(
    logging_directory="./data/logger",  # use subdirectory relative to Jupyter Notebook
    format_string="%(message)s"  # only show log messages, discard timestamp etc.
)

Creating log directory at ./data/logger

Creating log file at ./data/logger/2024-04-02--08-34-48.log

Database selection

Now, we select a database to work in.

database_chooser = conflowgen.DatabaseChooser(
    sqlite_databases_directory="./data/db"  # use subdirectory relative to Jupyter Notebook
)
demo_file_name = "my_demo.sqlite"

database_chooser.create_new_sqlite_database(demo_file_name, overwrite=True)

Creating SQLite directory at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/latest/docs/notebooks/data/db

No previous database detected, creating new at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/latest/docs/notebooks/data/db/my_demo.sqlite

Opening file /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/latest/docs/notebooks/data/db/my_demo.sqlite

journal_mode: wal

cache_size: -32768

page_size: 4096

foreign_keys: 1

Creating new database at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/latest/docs/notebooks/data/db/my_demo.sqlite

Creating all tables...

Seed with default values...

Number entries in table 'DeepSeaVessel': 0

Number entries in table 'Feeder': 0

Number entries in table 'Barge': 0

Number entries in table 'Train': 0

Number entries in table 'Truck': 0

Number entries in table 'Container': 0

Randomly set seed 1712046889178201126 for 'SqliteDatabaseConnection'

General settings

We use ContainerFlowGenerationManager.set_properties() to set a name, the start_date and the end_date for the generation process. The start_date and end_date parameters must be provided as a datetime.date. In our example, the end_time is set as a 21 day interval on top of the start_date which is set to the time of execution. Other general settings can be set here, see the ContainerFlowGenerationManager class for more information.

today = datetime.datetime.now().date()
container_flow_generation_manager = conflowgen.ContainerFlowGenerationManager()
container_flow_generation_manager.set_properties(
    name="Demo file",
    start_date=today,
    end_date=today + datetime.timedelta(days=21)
)

Randomly set seed 1712046889188918907 for 'TruckForImportContainersManager'

Randomly set seed 1712046889192123554 for 'TruckForExportContainersManager'

Randomly set seed 1712046889195259060 for 'ContainerFactory'

Randomly set seed 1712046889197974798 for 'LargeScheduledVehicleForOnwardTransportationManager'

Randomly set seed 1712046889210544376 for 'AllocateSpaceForContainersDeliveredByTruckService'

Replace seed 1712046889195259060 with 1712046889212959750 for 'ContainerFactory' for the new round.

Randomly set seed 1712046889215718942 for 'AssignDestinationToContainerService'

Loading destination distribution...

Creating schedules

In the next step, we add a feeder liner service to the database. We do this by using PortCallManager.add_vehicle(). The concrete vehicle instances are only generated when ContainerFlowGenerationManager.generate() is invoked.

port_call_manager = conflowgen.PortCallManager()

At first we define a name for our new feeder liner service.

feeder_service_name = "LX050"

By using PortCallManager.add_vehicle(), we can define the attributes for our feeder service.

• vehicle_type defines, that we deal with a feeder as the mode of transport. Other valid modes of transport are deep_sea_vessel, barge, or train.

• service_name defines a fictional name for that service.

• vehicle_arrives_at specifies the date where the first port call of this particular line is usually happening. This parameter must be a datetime.date.

• vehicle_arrives_at_time sets the scheduled timeslot of the port call. This parameter must be a datetime.time.

• average_vehicle_capacity defines the average capacity of the vessels utilized on this line. Parameter must be int or float.

• average_moved_capacity sets the capacity which is in average moved between the feeder and the terminal at each call. Parameter must be int or float.

• next_destinations can be set, consisting of name and frequency which can, e.g., be used as implication for storage and stacking problems. A list of name-frequency pairs is expected here.

port_call_manager.add_vehicle(
    vehicle_type=conflowgen.ModeOfTransport.feeder,
    service_name=feeder_service_name,
    vehicle_arrives_at=datetime.date(2021, 7, 9),
    vehicle_arrives_at_time=datetime.time(11),
    average_vehicle_capacity=800,
    average_moved_capacity=100,
    next_destinations=[
        ("DEBRV", 0.4),  # 40% of the containers go here...
        ("RULED", 0.6)   # and the other 60% of the containers go here.
    ]
)

Adding destination 'DEBRV' for schedule 'LX050' at the position 1 with a frequency of 40.00%

Adding destination 'RULED' for schedule 'LX050' at the position 2 with a frequency of 60.00%

Following the same principle and structure we can also add schedules for trains and deep sea vessels:

port_call_manager.add_vehicle(
    vehicle_type=conflowgen.ModeOfTransport.train,
    service_name="JR03A",
    vehicle_arrives_at=datetime.date(2021, 7, 12),
    vehicle_arrives_at_time=datetime.time(17),
    average_vehicle_capacity=90,
    average_moved_capacity=90,
    next_destinations=None  # Here we don't have containers that need to be grouped by destination
)

port_call_manager.add_vehicle(
    vehicle_type=conflowgen.ModeOfTransport.deep_sea_vessel,
    service_name="LX050",
    vehicle_arrives_at=datetime.date(2021, 7, 10),
    vehicle_arrives_at_time=datetime.time(19),
    average_vehicle_capacity=16000,
    average_moved_capacity=150,  # for faster demo
    next_destinations=[
        ("ZADUR", 0.3),  # 30% of the containers go to ZADUR...
        ("CNSHG", 0.7)   # and the other 70% of the containers go to CNSHG.
    ]
)

Adding destination 'ZADUR' for schedule 'LX050' at the position 1 with a frequency of 30.00%

Adding destination 'CNSHG' for schedule 'LX050' at the position 2 with a frequency of 70.00%

Generate the data

After we have set some exemplary schedules, we can now come to the actual generation. The generation process of the synthetic container flow data is started with ContainerFlowGenerationManager.generate(), based on the information you set earlier. With the optional parameter overwrite you determine the behavior if already some data exists - with the default behavior overwrite=True, any previous container flow data is overwritten, with overwrite=False the generation step is skipped if any data is detected.

container_flow_generation_manager.generate()

Resetting preview and analysis cache...

Remove previous data...

Reloading properties and distributions...

Using transportation buffer of 0.2 when choosing the departing vehicles that adhere a schedule.

Use transport buffer of 0.2 for allocating containers delivered by trucks.

For vehicle type truck and storage requirement empty, the container dwell times need to range from 0h to 1353h

For vehicle type truck and storage requirement standard, the container dwell times need to range from 0h to 511h

For vehicle type truck and storage requirement reefer, the container dwell times need to range from 0h to 511h

For vehicle type truck and storage requirement dangerous_goods, the container dwell times need to range from 0h to 511h

For vehicle type train and storage requirement empty, the container dwell times need to range from 0h to 684h

For vehicle type train and storage requirement standard, the container dwell times need to range from 0h to 208h

For vehicle type train and storage requirement reefer, the container dwell times need to range from 0h to 208h

For vehicle type train and storage requirement dangerous_goods, the container dwell times need to range from 0h to 208h

For vehicle type feeder and storage requirement empty, the container dwell times need to range from 3h to 979h

For vehicle type feeder and storage requirement standard, the container dwell times need to range from 3h to 223h

For vehicle type feeder and storage requirement reefer, the container dwell times need to range from 3h to 223h

For vehicle type feeder and storage requirement dangerous_goods, the container dwell times need to range from 3h to 223h

For vehicle type deep_sea_vessel and storage requirement empty, the container dwell times need to range from 3h to 878h

For vehicle type deep_sea_vessel and storage requirement standard, the container dwell times need to range from 3h to 216h

For vehicle type deep_sea_vessel and storage requirement reefer, the container dwell times need to range from 3h to 216h

For vehicle type deep_sea_vessel and storage requirement dangerous_goods, the container dwell times need to range from 3h to 216h

For vehicle type barge and storage requirement empty, the container dwell times need to range from 0h to 691h

For vehicle type barge and storage requirement standard, the container dwell times need to range from 0h to 576h

For vehicle type barge and storage requirement reefer, the container dwell times need to range from 0h to 576h

For vehicle type barge and storage requirement dangerous_goods, the container dwell times need to range from 0h to 576h

For vehicle type truck and storage requirement empty, the container dwell times need to range from 0h to 1353h

For vehicle type truck and storage requirement standard, the container dwell times need to range from 0h to 511h

For vehicle type truck and storage requirement reefer, the container dwell times need to range from 0h to 511h

For vehicle type truck and storage requirement dangerous_goods, the container dwell times need to range from 0h to 511h

For vehicle type train and storage requirement empty, the container dwell times need to range from 0h to 828h

For vehicle type train and storage requirement standard, the container dwell times need to range from 0h to 122h

For vehicle type train and storage requirement reefer, the container dwell times need to range from 0h to 122h

For vehicle type train and storage requirement dangerous_goods, the container dwell times need to range from 0h to 122h

For vehicle type feeder and storage requirement empty, the container dwell times need to range from 12h to 936h

For vehicle type feeder and storage requirement standard, the container dwell times need to range from 12h to 257h

For vehicle type feeder and storage requirement reefer, the container dwell times need to range from 12h to 257h

For vehicle type feeder and storage requirement dangerous_goods, the container dwell times need to range from 12h to 257h

For vehicle type deep_sea_vessel and storage requirement empty, the container dwell times need to range from 12h to 964h

For vehicle type deep_sea_vessel and storage requirement standard, the container dwell times need to range from 12h to 468h

For vehicle type deep_sea_vessel and storage requirement reefer, the container dwell times need to range from 12h to 468h

For vehicle type deep_sea_vessel and storage requirement dangerous_goods, the container dwell times need to range from 12h to 468h

For vehicle type barge and storage requirement empty, the container dwell times need to range from 0h to 532h

For vehicle type barge and storage requirement standard, the container dwell times need to range from 0h to 720h

For vehicle type barge and storage requirement reefer, the container dwell times need to range from 0h to 720h

For vehicle type barge and storage requirement dangerous_goods, the container dwell times need to range from 0h to 720h

Loading destination distribution...

Load destination distribution for service 'LX050' by feeder with 2 destinations

Load destination distribution for service 'LX050' by deep_sea_vessel with 2 destinations

Create fleet including their delivered containers for given time range for each schedule...

Create vehicles and containers for service 'JR03A' of type 'train', progress: 1 / 3 (33.33%)

Create vehicles and containers for service 'LX050' of type 'deep_sea_vessel', progress: 2 / 3 (66.67%)

Create vehicles and containers for service 'LX050' of type 'feeder', progress: 3 / 3 (100.00%)

Loading status of vehicles adhering to a schedule:

Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.67
Stddev:  0.67
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 90.00
Maximum: 180.00
Mean:    130.00
Stddev:  39.69
(rounding errors might exist)

Assign containers that are picked up from the terminal by a vehicle adhering a schedule to their specific vehicle instance...

Assign containers to departing vehicles that move according to a schedule...

In total, 524 containers continue their journey on a vehicle that adhere to a schedule, assigning these containers to their respective vehicles...

Progress: 1 / 524 (0.19%) containers have been assigned to a scheduled vehicle to leave the terminal again.

Progress: 524 / 524 (100.00%) containers have been assigned to a scheduled vehicle to leave the terminal again.

Containers for which no outgoing vehicle adhering to a schedule could be found: 77.29%. These will be re-assigned to another vehicle type, such as a truck.

All containers for which a departing vehicle that moves according to a schedule was available have been assigned to one.

Loading status of vehicles adhering to a schedule:

Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.67
Stddev:  0.67
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 0.50
Maximum: 178.00
Mean:    53.39
Stddev:  59.19
(rounding errors might exist)

Generate trucks that pick up containers...

In total 193 containers are picked up by truck, creating these trucks now...

Progress: 1 / 193 (0.52%) trucks generated to pick up containers at the terminal.

Progress: 193 / 193 (100.00%) trucks generated to pick up containers at the terminal.

All 193 trucks that pick up a container have been generated, moving 315.5 TEU.

Generate containers that are delivered by trucks...

Progress: 1 / 108 (0.93%) of the containers which are delivered by truck are allocated on a vehicle adhering to a schedule

Progress: 108 / 108 (100.00%) of the containers which are delivered by truck are allocated on a vehicle adhering to a schedule

All 108 containers that need to be delivered by truck have been assigned to a vehicle that adheres to a schedule, corresponding to 168.5 TEU.

Loading status of vehicles adhering to a schedule:

Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.67
Stddev:  0.67
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 0.50
Maximum: 97.50
Mean:    34.67
Stddev:  33.76
(rounding errors might exist)

Generate trucks that deliver containers...

In total 108 containers are delivered by truck, creating these trucks now...

Progress: 1 / 108 (0.93%) trucks generated to deliver containers to the terminal.

Progress: 108 / 108 (100.00%) trucks generated to deliver containers to the terminal.

All 108 trucks that deliver a container are created now, moving 168.5 TEU.

Assign containers to next destinations...

Assign destinations to containers that leave the terminal with the service 'LX050' of the vehicle type feeder, progress: 1 / 2 (50.00%)

Assign destinations to containers that leave the terminal with the service 'LX050' of the vehicle type deep_sea_vessel, progress: 2 / 2 (100.00%)

Container flow generation finished

Final capacity status of vehicles adhering to a schedule:

Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.67
Stddev:  0.67
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 0.50
Maximum: 97.50
Mean:    34.67
Stddev:  33.76
(rounding errors might exist)

Export the data

The data is exported with two lines of code.

export_container_flow_manager = conflowgen.ExportContainerFlowManager()
path_to_exported_data = export_container_flow_manager.export("first_steps", "./data/export", overwrite=True)

Creating export folder ./data/export

Creating folder at ./data/export/first_steps

Converting SQL database into file format '.csv'

Resolving column destination of model <Model: Container>...

This is a nested call to resolve the column 'destination'

Gathering data for generating the 'deep_sea_vessels' table...

Resolving column large_scheduled_vehicle of model <Model: DeepSeaVessel>...

This is a nested call to resolve the column 'large_scheduled_vehicle'

Gathering data for generating the 'feeders' table...

Resolving column large_scheduled_vehicle of model <Model: Feeder>...

Gathering data for generating the 'barges' table...

No content found for the barges table, the file will be empty.

Gathering data for generating the 'trains' table...

Resolving column large_scheduled_vehicle of model <Model: Train>...

Resolving column truck_arrival_information_for_pickup of model <Model: Truck>...

This is a nested call to resolve the column 'truck_arrival_information_for_pickup'

Resolving column truck_arrival_information_for_delivery of model <Model: Truck>...

This is a nested call to resolve the column 'truck_arrival_information_for_delivery'

Saving file containers.csv

Saving file deep_sea_vessels.csv

Saving file feeders.csv

Saving file barges.csv

Saving file trains.csv

Saving file trucks.csv

Export has finished successfully.

Examining the exported data

The CSV files we can open e.g. with pandas. Here, the tabular data is presented.

df_containers = pd.read_csv(
    os.path.join(path_to_exported_data, "containers.csv"),
    index_col="id",
    dtype={
        "delivered_by_truck": "Int64",
        "picked_up_by_truck": "Int64",
        "delivered_by_vehicle": "Int64",
        "picked_up_by_vehicle": "Int64",
        "destination_sequence_id": "Int64"
    }
)
df_containers

	weight	length	storage_requirement	delivered_by	picked_up_by_initial	picked_up_by	delivered_by_vehicle	delivered_by_truck	picked_up_by_vehicle	picked_up_by_truck	emergency_pickup	destination_sequence_id	destination_name
id
1	12	40	standard	train	deep_sea_vessel	deep_sea_vessel	1	<NA>	5	<NA>	False	2	CNSHG
2	26	20	dangerous_goods	train	deep_sea_vessel	deep_sea_vessel	1	<NA>	5	<NA>	False	2	CNSHG
3	14	40	standard	train	feeder	feeder	1	<NA>	8	<NA>	False	2	RULED
4	14	45	standard	train	deep_sea_vessel	deep_sea_vessel	1	<NA>	5	<NA>	False	1	ZADUR
5	4	40	empty	train	deep_sea_vessel	deep_sea_vessel	1	<NA>	6	<NA>	False	2	CNSHG
...	...	...	...	...	...	...	...	...	...	...	...	...	...
736	14	45	standard	truck	deep_sea_vessel	deep_sea_vessel	<NA>	247	4	<NA>	False	2	CNSHG
737	4	40	empty	truck	deep_sea_vessel	deep_sea_vessel	<NA>	233	4	<NA>	False	2	CNSHG
738	10	40	standard	truck	deep_sea_vessel	deep_sea_vessel	<NA>	265	5	<NA>	False	2	CNSHG
739	20	20	standard	truck	deep_sea_vessel	deep_sea_vessel	<NA>	277	4	<NA>	False	2	CNSHG
740	10	40	standard	truck	deep_sea_vessel	deep_sea_vessel	<NA>	211	4	<NA>	False	1	ZADUR

740 rows × 13 columns

The column delivered_by contains the vehicle type a container is delivered by. For trucks, the column delivered_by_truck mentions the corresponding ID of the vehicle that is stored in the separate table trucks.csv. The same scheme applies to picked_up_by and picked_up_by_truck.

df_trucks = pd.read_csv(
    os.path.join(path_to_exported_data, "trucks.csv"),
    index_col="id"
)
df_trucks

	delivers_container	picks_up_container	realized_container_pickup_time	realized_container_delivery_time
id
1	False	True	2024-04-22 06:56:36.925425	NaN
2	False	True	2024-04-16 06:37:48.614231	NaN
3	False	True	2024-04-15 07:55:22.111579	NaN
4	False	True	2024-04-15 11:37:40.218494	NaN
5	False	True	2024-04-12 13:30:23.518964	NaN
...	...	...	...	...
297	True	False	NaN	2024-03-29 20:46:06.787329
298	True	False	NaN	2024-04-03 20:33:07.519233
299	True	False	NaN	2024-04-18 07:08:35.247171
300	True	False	NaN	2024-04-08 08:45:00.134501
301	True	False	NaN	2024-04-02 06:27:47.550206

301 rows × 4 columns

Here, we can see which trucks deliver a container, pick up a container, and for which time their arrivals are realized.

All vehicle types except trucks are considered vehicles that typically arrive according to a schedule and which move larger amounts of containers at once. Thus, the table has a slightly different shape. They all have ids from the same ID range and for each vehicle type the ids might be non-consecutive. So e.g. for trains, in the column delivered_by_vehicle in the container table the ID is mentioned that we can look up in the table of trains.csv.

df_trains = pd.read_csv(
    os.path.join(path_to_exported_data, "trains.csv"),
    index_col="id"
)
df_trains

	vehicle_name	capacity_in_teu	moved_capacity	realized_arrival
id
1	1	90	90	2024-04-08 17:00:00
2	2	90	90	2024-04-15 17:00:00
3	3	90	90	2024-04-22 17:00:00

Corresponding CSV files exist for the other vehicles as well.