Skip to content

Task5 Implementation

Now we will see the implementation of Task 5 For Better understanding of Task 5 implementation we have divided the task into few sub tasks

Sub Task 1 (Colour Detection)

For colour detection or object detection we are using QR decoding.First we take data from 2D camera from the topic /eyrc/vb/camera_1/image_raw and then we clean the image using thresolding(cv2.threshold( )) further we decode it into useful information.The QR decoder returns few arguments like data, type, rect and polygon.In our task implementation we are using rect and data in which rect returns four arguments x,y,w,h where x,y are the corner pixel values while w is width and h is height of the detected box.We are using these pixel values to determine the postion and colour of the box.To use this information further we encoded it like this in two different ways:-

box_color=[]
for i in qr_result:
    x,y,h,w=i.rect
    if (x>100 and x<200) and (y>250 and y<400):
            box_color.append(('packagen00',i.data))
    ......

box_color=[(packagen00,red),(packagen01,yellow),(packagen02,green).......and so on]

box_color=[]                                        
for i in qr_result:
    x,y,h,w=i.rect
    if (x>100 and x<200) and (y>250 and y<400):
            box_color.append(('R0 C0',i.data))
    .......

box_color=[(R0 C0,red),(R0 C1,yellow),(R0 C2,green).......and so on]

Sub Task 2 (Pushing Data on Inventory and Incoming order Spreadsheet)

By the QR detection done in sub task 1 we made an array of tuple.We then sorted data by looping over this array of tuple using package colour and package position.Using this data we updated the inventory spreadsheet.

For taking the online incoming order we subscribe the mqtt topic called as /eyrc/vb/VsMyPsSr/orders and after sorting the data (we got by subscribing the mqtt topic) is then updated on the incoming oredrs spreadsheet.

Sub Task 3 (Priority Check and order disptach)

For priority check we first subscribe the topic /eyrc/vb/VsMyPsSr/orders in the node ur5_1.py that takes the online order,then we put the order_id in a list which is defined globally.

#This is the callback function for topic '/eyrc/vb/VsMyPsSr/orders'
def mqtt_orders_1(self, data):
    global lst, count,timer
    order_list = data.message
    goal_message=(order_list.split(", "))
    item = (goal_message[5])[9:len(goal_message[5])-1]
    order_id1 = (goal_message[2])[13:len(goal_message[2])-1]
    order.append((order_id1,order_list))
    lst.append(order_id1)

After that we subscribe a new topic /eyrc/vb/camera_1/image_raw that works parallel with the previous topic. The main motive of this topic is to take the list of orders from the online order topic and operate on the list("") of data simultaneously. While processing the orders there may come a time when we can have multiple orders of different priorities to process this is where we check the priority of the order.We have used order_id as a key to sort the orders in high to low priority.As the oredr id is a string so we use sort( ) function for sorting.By doing this we make sure that High Priority orders are processed as quickly as possible, then Medium Priority and then Low Priority orders are processed. 

#This is the callback function for topic '/eyrc/vb/camera_1/image_raw'
def mqtt_orders_2(self, data):
    global lst
    lst.sort()

    while lst:
        lst.sort()
        order_id=lst[0]
        if(order_id=='3001'):
            self.packagen02_box('3001') #This will pic the order and place it on conveyer
            self.msg_order_1(order_list) #updating order in OrdersDispatch spreadsheet
            lst.remove(order_id)
        elif(order_id=='3002'):
            self.packagen10_box('3002')
            self.msg_order_1(order_list)
            lst.remove(order_id)
        elif(order_id=='3003'):
            self.packagen20_box('3003')
            self.msg_order_1(order_list)
            lst.remove(order_id)
        ............
        .....and so on

Frther we Dispatch (pick the package from shelf and place it on the conveyer) the order using ur5_1 arm and then we update the OrdersDispatch spreadsheet.

Sub Task 4 (Ur5_2 and Shipping Orders)

As we have encoded the data in sub task 1 from the topic (eyrc/vb/camera_1/image_raw).This data is stored in an array. Now we subscribe the logical_camera_2 topic (/eyrc/vb/logical_camera_2).The callback of this topic gives the model name and position in the frame of logical camera when any object comes under this logical_camera_2. Using the positon of the incoming model we stop the conveyer belt and then using the data we encoded in subtask 1 we loop over that array then we match the model name with package name and identify the colour of the package then we Ship (put them into the bins according to the colour of the package) the order using ur5_2 arm and update the OrdersShipped Spreadsheet.

for i in reversed(list_1):
    model = data.models[1].type
    pkg_n,pkg_clr=i

    if(pkg_clr=='red' and model==pkg_n):
        if y>-0.03 and y<0.02:           #This condition will stop the conveyer belt
            self.conveyor(0)
            self.pkg1(x,y,z)             #This will place the package from conveyer to bin

    elif(pkg_clr=='yellow' and model==pkg_n):
        if  y>-0.03 and y<0.03:
            self.conveyor(0)
            self.pkg2(x,y,z)

    elif(pkg_clr=='green' and model==pkg_n):
        if  y>-0.04 and y<0.03:
            self.conveyor(0)
            self.pkg3(x,y,z)
        ............
        .....and so on

Sub Task 5 (Dashborad)

After properly implementing the above four sub tasks and updating the IncomingOrders ,OredrsDispatch & OrdersShipped sheets using these three sheets we now update the Dashboard sheet of the Inventory management Mastersheet which will be further used in the Dashborad web page that we made using Github Pages.Using Github pages we made a dynamic dashboard which includes a map which tells the location of the incoming orders a graph which tells total time taken to ship a package and a table which shows all the necessary informtions like Order ID ,Item ,Priority , Quantity, City , Dispatched ,Shipped , Order Date and Time , Dispatch Date and Time , Shipped Date and Time , Time Taken.

Now to update our webpage using values from Dashboard sheet we use that sheet as a JSON endpoint which gives us the entire content of the sheet in a JSON format which is then used in our webpage. Then we update the JSON data using AJAX( Asynchronous JavaScript and XML) request which makes requests to the server without reloading the page , receive and work with data from the server so that we dont have to refresh the webpage.

Task5 implemetation video