First steps

ConFlowGen has been developed with Jupyter Notebooks in mind. While it is also possible to write your code as pure Python Scripts (see e.g. the Python Script examples), Jupyter Notebooks allow to mix the code with explanatory texts, LaTeX formulas, etc. in a single document. In addition, the generated visualisations neatly blend in.

For newcomers it is suggested to download the Anaconda distribution and first to familiarize yourself with the newly installed tools. Then, it is suggested to create a separate environment as described for the Jupyter Notebook examples. During the environment set up, ConFlowGen is automatically installed. Then, you can start JupyterLab from Anaconda Navigator or from the CLI. Please ensure that you are in the correct conda environment (it should be conflowgen) when you work.

Prerequisites

To start, we first import ConFlowGen as a module.

import os
import datetime

import pandas as pd

import conflowgen

Then, we set up the logger

logger = conflowgen.setup_logger(
    logging_directory="./data/logger",  # use subdirectory relative to Jupyter Notebook
    format_string="%(message)s"  # only show log messages, discard timestamp etc.
)

Creating log directory at ./data/logger
Creating log file at ./data/logger/2022-09-14--10-44-34.log

Database selection

Now, we select a database to work in.

database_chooser = conflowgen.DatabaseChooser(
    sqlite_databases_directory="./data/db"  # use subdirectory relative to Jupyter Notebook
)
demo_file_name = "my_demo.sqlite"

database_chooser.create_new_sqlite_database(demo_file_name, overwrite=True)

Creating SQLite directory at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/v1.0.4/docs/notebooks/data/db
No previous database detected, creating new at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/v1.0.4/docs/notebooks/data/db/my_demo.sqlite
Opening file /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/v1.0.4/docs/notebooks/data/db/my_demo.sqlite
journal_mode: wal
cache_size: -32768
page_size: 4096
foreign_keys: 1
Creating new database at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/v1.0.4/docs/notebooks/data/db/my_demo.sqlite
Creating all tables...
Seed with default values...

General settings

We use ContainerFlowGenerationManager.set_properties() to set a name, the start_date and the end_date for the generation process. The start_date and end_date parameters must be provided as a datetime.date. In our example, the end_time is set as a 21 day interval on top of the start_date which is set to the time of execution. Other general settings can be set here, see the ContainerFlowGenerationManager class for more information.

container_flow_generation_manager = conflowgen.ContainerFlowGenerationManager()
container_flow_generation_manager.set_properties(
    name="Demo file",
    start_date=datetime.datetime.now().date(),
    end_date=datetime.datetime.now().date() + datetime.timedelta(days=21)
)

Loading destination distribution...

Creating schedules

In the next step, we add a feeder liner service to the database. We do this by using PortCallManager.add_large_scheduled_vehicle(). The concrete vehicle instances are only generated when ContainerFlowGenerationManager.generate() is invoked.

port_call_manager = conflowgen.PortCallManager()

At first we define a name for our new feeder liner service.

feeder_service_name = "LX050"

By using PortCallManager.add_large_scheduled_vehicle(), we can define the attributes for our feeder service.

• vehicle_type defines, that we deal with a feeder as the mode of transport. Other valid modes of transport are deep_sea_vessel, barge, or train.

• service_name defines a fictional name for that service.

• vehicle_arrives_at specifies the date where the first port call of this particular line is usually happening. This parameter must be a datetime.date.

• vehicle_arrives_at_time sets the average / planned / scheduled timeslot of the port call. This parameter must be a datetime.time.

• average_vehicle_capacity defines the average capacity of the vessels utilized on this line. Parameter must be int or float.

• average_moved_capacity sets the capacity which is in average moved between the feeder and the terminal at each call. Parameter must be int or float.

• next_destinations can be set, consisting of name and frequency which can e.g. be used as implication for storage and stacking problems. A list of tuples [str, float] is expected here.

port_call_manager.add_large_scheduled_vehicle(
    vehicle_type=conflowgen.ModeOfTransport.feeder,
    service_name=feeder_service_name,
    vehicle_arrives_at=datetime.date(2021, 7, 9),
    vehicle_arrives_at_time=datetime.time(11),
    average_vehicle_capacity=800,
    average_moved_capacity=100,
    next_destinations=[
        ("DEBRV", 0.4),  # 40% of the containers go here...
        ("RULED", 0.6)   # and the other 60% of the containers go here.
    ]
)

Adding destination 'DEBRV' for schedule 'LX050' at the position 1 with a frequency of 40.00%
Adding destination 'RULED' for schedule 'LX050' at the position 2 with a frequency of 60.00%

Following the same principle and structure we can also add schedules for trains and deep sea vessels:

port_call_manager.add_large_scheduled_vehicle(
    vehicle_type=conflowgen.ModeOfTransport.train,
    service_name="JR03A",
    vehicle_arrives_at=datetime.date(2021, 7, 12),
    vehicle_arrives_at_time=datetime.time(17),
    average_vehicle_capacity=90,
    average_moved_capacity=90,
    next_destinations=None  # Here we don't have containers that need to be grouped by destination
)

port_call_manager.add_large_scheduled_vehicle(
    vehicle_type=conflowgen.ModeOfTransport.deep_sea_vessel,
    service_name="LX050",
    vehicle_arrives_at=datetime.date(2021, 7, 10),
    vehicle_arrives_at_time=datetime.time(19),
    average_vehicle_capacity=16000,
    average_moved_capacity=150,  # for faster demo
    next_destinations=[
        ("ZADUR", 0.3),  # 30% of the containers go to ZADUR...
        ("CNSHG", 0.7)   # and the other 70% of the containers go to CNSHG.
    ]
)

Adding destination 'ZADUR' for schedule 'LX050' at the position 1 with a frequency of 30.00%
Adding destination 'CNSHG' for schedule 'LX050' at the position 2 with a frequency of 70.00%

Generate the data

After we have set some exemplary schedules, we can now come to the actual generation. The generation process of the synthetic container flow data is started with ContainerFlowGenerationManager.generate(), based on the information you set earlier. With the optional parameter overwrite you determine the behavior if already some data exists - with the default behavior overwrite=True, any previous container flow data is overwritten, with overwrite=False the generation step is skipped if any data is detected.

container_flow_generation_manager.generate()

Remove previous data...
Reloading properties and distributions...
Using transportation buffer of 0.2 when choosing the departing vehicles that adhere a schedule.
Use transport buffer of 0.2 for allocating containers delivered by trucks
Loading destination distribution...
Load destination distribution for service 'LX050' by feeder
Destination 'DEBRV' is frequented by 40.00% of the containers and is number 1
Destination 'RULED' is frequented by 60.00% of the containers and is number 2
Load destination distribution for service 'LX050' by deep_sea_vessel
Destination 'ZADUR' is frequented by 30.00% of the containers and is number 1
Destination 'CNSHG' is frequented by 70.00% of the containers and is number 2
Create fleet including their delivered containers for given time range for each schedule...
Create vehicles and containers for service 'LX050' of type 'feeder', progress: 1 / 3 (33.33%)
Create vehicles and containers for service 'JR03A' of type 'train', progress: 2 / 3 (66.67%)
Create vehicles and containers for service 'LX050' of type 'deep_sea_vessel', progress: 3 / 3 (100.00%)
Loading status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.83
Stddev:  0.70
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 90.00
Maximum: 180.00
Mean:    130.00
Stddev:  39.69
(rounding errors might exist)

Assign containers arriving by vehicles adhering a schedule for onward transportation...
Assign containers to departing vehicles that move according to a schedule...
In total 493 containers continue their journey on a vehicle that adhere to a schedule, assigning these containers to their respective vehicles...
All containers for which a departing vehicle that moves according to a schedule was available have been assigned to one.
Containers for which no outgoing vehicle could be found: 60.24%
Loading status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.83
Stddev:  0.70
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 0.00
Maximum: 120.00
Mean:    55.03
Stddev:  49.49
(rounding errors might exist)

Generate trucks that pick up containers...
In total 199 containers are picked up by truck, creating these trucks now...
All trucks that pick up a container have been generated.
Generate containers that are delivered by trucks...
Vehicle 'LX050_feeder_2' of type 'feeder' has no remaining capacity and is no further tried - free capacity of 0.75 TEU is less than the required 2.5 TEU.
All containers that need to be delivered by truck have been assigned to a vehicle that moves according to a schedule.
Loading status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.83
Stddev:  0.70
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 0.00
Maximum: 40.00
Mean:    10.81
Stddev:  15.69
(rounding errors might exist)

Generate trucks that deliver containers...
In total 199 containers are delivered by truck, creating these trucks now...
All trucks that deliver a container are created now.
Assign containers to next destinations...
Assign destinations to containers that leave the terminal with the service 'LX050' of the vehicle type feeder, progress: 1 / 2 (50.00%)
Assign destinations to containers that leave the terminal with the service 'LX050' of the vehicle type deep_sea_vessel, progress: 2 / 2 (100.00%)
Container flow generation finished
Final capacity status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.

Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean:    1.83
Stddev:  0.70
(rounding errors might exist)

Free Outbound Capacity Statistics
Minimum: 0.00
Maximum: 40.00
Mean:    10.81
Stddev:  15.69
(rounding errors might exist)

Export the data

The data is exported with two lines of code.

export_container_flow_manager = conflowgen.ExportContainerFlowManager()
path_to_exported_data = export_container_flow_manager.export("first_steps", "./data/export", overwrite=True)

Creating export folder ./data/export
Creating folder at ./data/export/first_steps
Converting SQL database into file format '.csv'
Resolving column destination of model <Model: Container>...
This is a nested call to resolve the column 'destination'
Gathering data for generating the 'deep_sea_vessels' table...
Resolving column large_scheduled_vehicle of model <Model: DeepSeaVessel>...
This is a nested call to resolve the column 'large_scheduled_vehicle'
Gathering data for generating the 'feeders' table...
Resolving column large_scheduled_vehicle of model <Model: Feeder>...
Gathering data for generating the 'barges' table...
No content found for the barges table, the file will be empty.
Gathering data for generating the 'trains' table...
Resolving column large_scheduled_vehicle of model <Model: Train>...
Resolving column truck_arrival_information_for_pickup of model <Model: Truck>...
This is a nested call to resolve the column 'truck_arrival_information_for_pickup'
Resolving column truck_arrival_information_for_delivery of model <Model: Truck>...
This is a nested call to resolve the column 'truck_arrival_information_for_delivery'
Saving file containers.csv
Saving file deep_sea_vessels.csv
Saving file feeders.csv
Saving file barges.csv
Saving file trains.csv
Saving file trucks.csv
Export has finished successfully.

Examining the exported data

The CSV files we can open e.g. with pandas. Here, the tabular data is presented.

df_containers = pd.read_csv(
    os.path.join(path_to_exported_data, "containers.csv"),
    index_col="id",
    dtype={
        "delivered_by_truck": "Int64",
        "picked_up_by_truck": "Int64",
        "delivered_by_large_scheduled_vehicle": "Int64",
        "picked_up_by_large_scheduled_vehicle": "Int64",
        "destination_sequence_id": "Int64"
    }
)
df_containers

	weight	length	storage_requirement	delivered_by	picked_up_by_initial	picked_up_by	delivered_by_vehicle	delivered_by_truck	picked_up_by_vehicle	picked_up_by_truck	emergency_pickup	destination_sequence_id	destination_name
id
1	6	40	standard	feeder	train	deep_sea_vessel	1.0	<NA>	7.0	<NA>	True	2	CNSHG
2	2	20	empty	feeder	deep_sea_vessel	deep_sea_vessel	1.0	<NA>	7.0	<NA>	False	2	CNSHG
3	4	45	empty	feeder	deep_sea_vessel	deep_sea_vessel	1.0	<NA>	7.0	<NA>	False	2	CNSHG
4	24	20	standard	feeder	deep_sea_vessel	deep_sea_vessel	1.0	<NA>	7.0	<NA>	False	2	CNSHG
5	18	40	standard	feeder	train	deep_sea_vessel	1.0	<NA>	7.0	<NA>	True	2	CNSHG
...	...	...	...	...	...	...	...	...	...	...	...	...	...
812	14	40	standard	truck	feeder	feeder	NaN	394	1.0	<NA>	False	2	RULED
813	10	40	reefer	truck	deep_sea_vessel	deep_sea_vessel	NaN	395	7.0	<NA>	False	2	CNSHG
814	12	40	standard	truck	deep_sea_vessel	deep_sea_vessel	NaN	396	7.0	<NA>	False	2	CNSHG
815	12	40	standard	truck	deep_sea_vessel	deep_sea_vessel	NaN	397	7.0	<NA>	False	2	CNSHG
816	14	40	standard	truck	deep_sea_vessel	deep_sea_vessel	NaN	398	8.0	<NA>	False	2	CNSHG

816 rows × 13 columns

The column delivered_by contains the vehicle type a container is delivered by. For trucks, the column delivered_by_truck mentions the corresponding ID of the vehicle that is stored in the separate table trucks.csv. The same scheme applies to picked_up_by and picked_up_by_truck.

df_trucks = pd.read_csv(
    os.path.join(path_to_exported_data, "trucks.csv"),
    index_col="id"
)
df_trucks

	delivers_container	picks_up_container	realized_container_pickup_time	realized_container_delivery_time
id
1	False	True	2022-09-19 06:19:27.882096	NaN
2	False	True	2022-09-19 06:25:12.568187	NaN
3	False	True	2022-09-16 20:15:23.601136	NaN
4	False	True	2022-09-19 10:42:32.693755	NaN
5	False	True	2022-09-16 19:10:45.284557	NaN
...	...	...	...	...
394	True	False	NaN	2022-09-12 16:06:39.348985
395	True	False	NaN	2022-09-16 09:36:10.389006
396	True	False	NaN	2022-09-15 13:32:13.926540
397	True	False	NaN	2022-09-15 21:31:35.695681
398	True	False	NaN	2022-09-20 16:04:00.040974

398 rows × 4 columns

Here, we can see which trucks deliver a container, pick up a container, and for which time their arrivals are realized.

All vehicle types except trucks are considered vehicles that typically arrive according to a schedule and which move larger amounts of containers at once. Thus, the table has a slightly different shape. They all have ids from the same ID range and for each vehicle type the ids might be non-consecutive. So e.g. for trains, in the column delivered_by_vehicle in the container table the ID is mentioned that we can look up in the table of trains.csv.

df_trains = pd.read_csv(
    os.path.join(path_to_exported_data, "trains.csv"),
    index_col="id"
)
df_trains

	vehicle_name	capacity_in_teu	moved_capacity	realized_arrival
id
4	JR03A_train_1	90	90	2022-09-19 17:00:00
5	JR03A_train_2	90	90	2022-09-26 17:00:00
6	JR03A_train_3	90	90	2022-10-03 17:00:00

Corresponding CSV files exist for the other vehicles as well.