Minutes 28.06.2021 Discussion next steps

Participants: Balakrishna (Fokus), Kurt, Heinz (Linutronix), Sven, Ivan, JP (Nexedi)

Agenda:

• Welcome
• Update from everyone
• Discussion on late deliverables
• Discussion on points to focus on during next 6 months
• Other topics

Discussion:

• S: recap of progress from Tsvetan regarding new board - new board for AI processing with camera inputs by September/October, progress on robot, can now solder, next step will be to add the ML/AI required hardware/software
• K: setup software TSN, fixed bugs in Linux, now extending architecture with more devices to test, results promising
• I: replaced Proview with Beremiz, everything working and deployed with SlapOS, old protocol finished
• B: worked on OPC-UA over TCP and HTTP using JSON
• S: main blocker for next deliverables will be SlapOS protocol, connecting everything will depend on that
• I: will take at least 6 months
• S: except Fokus, everyone can work relatively autonomous and at some point we need to connect everything together. 
• JP: Bala can help with standard writing, exchange later
• S: no other questions? Closing

Minutes 18.04.2021 Discussion Linutronix next steps

Participants: Heinz Egger, Kurt Kanzenbach, Jan Altenberg (LTX), Jean-Paul Smets, Sven Franck (Nexedi)

Agenda:

1. Discussion next steps

Discussion:

• H: real time extension of SlapOS and OPC-UA, not for us, we are doing software development on jitter
• JP: turning SlapOS into something reproducable by 3rd party, this is what it does in each case, we have positive cases of Slap protocol. We have pure REST implementation, next Romain, Jean-Paul and Cedric we will refine the protocol. In reality you don't care. We care a lot, because we want a simple fully-specified protocol, so we can compete with Kubernetes, becomes essential thing for me. Public funds in DE and FR used to sponsor destruction of existing technologies. I want to have a standard, that can be pushed to standardisation body.
• JP: drawing from last time - demo of guaranteed latency between two nodes (...), is it finished, do we have figures for latency
• H: not done, software development
• JP: PubSub over routed network + security, if we consider IA, we only need PubSub to implement a topology, we want every node to get a keep-alive from every part (I'm alive, I'm alive, I'm alive...). We'd like to see that implemented with OPC-UA PubSub.
• H: objective?
• JP: we plan to use TSN WLAN to implement keep-alive between Master and Leaf - safety for IA system. If this is a koppler and central virtual PLC. PLC will use non-real-time WLAN to send/receive events. but we need safety network with guaranteed latency, every leaf says, I'm alive...., if master misses one leaf, go to safe mode, if leaf doesn't get alive from master, safe mode
• H: TSN community talk about different time zones, at least two different time zones, they get safety by comparing time zones, this is not yet solved for TSN router problem.
• JP: in PTP, master sends time to slaves. Slaves are not sending stuff. 
• H: yep, but some where you have time receiving from two different masters, that allows to see if one master has stopped or not
• JP: Is it a problem to implmement heartbeat model. Keep alive over TSN for OSIE, we can survive without routing for TSN, we already know how to set all adresses. Using OPC-UA PubSub for keepalive in both ways, not too difficult.
• H: get the picture
• JP: OSIE star case, Dronedge mesh case
• JP: airplanes send message every x sec what they do and where they are
• H: transponder/responder, blackbox, same for chips
• JP: Drone will emit, I'm here, this is what I do. We want all drones to subscribe to these messages. Same as previous case, WLAN for high-priority transponder/responder, another WLAN for general messages. We will try to split communication on two different routes to minimize congestion, also using OPC-UA PubSub, so drones can subscribe to other drones. So instead of keep-alive in star topologigy, this is more of a keep-alive in a graph topology
• JP: If we look at previous meeting we still talk about problem 5, which is anyway a good idea to implement. For me, we need a kind a prototype to play with, that shows "I'm alive" or "I'm here"
• JP: (...) I'm alive with guranteed latency is mainly a problem with strict TSN, I'm here with minimal latency is just queuing policies (because here we have good/bad radio)
• JP: transponder/responder could be implemented with OPC-UA PubSub, transponder/responder is not really TSN, but OPC-UA PubSub would be perfect. This way technology is more or less the same
• H: Ok
• JP: very useful contribution from Linutronix for us. Everything else we now know how to do,
• J: OSIE clear, working on strict latency, we investigate problems with Linux and how to ensure to get bounded latency
• JP: explain, general network with different links, blue,yellow:broadcast, only communication possible where there is a link.Radio system it represents a neighbour who can be reached
• JP: 1st problem to solve: How can we find the shortest route between two points based on latency, solved with Babel RFC....
• JP: 2nd sync clock of every node? To sync clock between drones we use GPS, we use PTP in IA. We worked about a year ago, LTX taught us how to sync nodes using PTP, problem solved
• JP: 3rd guaranteed latency for certain types of communication between two nodes. Create two WLAN, different policies, one will have guaranteed latency, prototyped by LTX in January 2021
• Jan: idea of TSN is to use link and divide into timeslots and give priorities
• JP: this is what you're working on. Base technology on which to build the rest
• JP: how to map guaranteed time slots on a link to an IPv6 adress space. Suppose every node has one IPv6 adress for general purpose communication and second for guarateed communication. Use this adress or that adress
• Jan: need to look into how to map in Linux
• JP: can be done by routing protocol, source specific routing, technology implemented by babel protocol, too
• Jan: from Linux point of view, application decides which times slots to use not based on IP adresse but socket priority
• JP: we map latency based on IP adresses
• Jan: possible with some small extension
• JP: source specific routing is a fantastic idea, any type of management of quality of service network topology with tags is isomorphic to source specific routing (?), only chose which adress to bind to. This part of generating routes, we know how to do
• JP: problem 5: discussing today, how can we do real-time PubSub in two cases: every leaf is subscribing to master and master subscribes to all leafs, done in a way that the PubSub itself is in real-time
• JP: did I reply to your question
• H: need to think about the transponder/responder
• Jan: OPC-UA PubSub either works on Eithernet/Mulitcast or based on UDP using Multicast, will only work between two neighbouring objects. 
• JP: UDP multicast should work, else we ...
• Jan: not sure it work with OPC-UA PubSub or we need to modify
• H: mentioned last time
• JP: usually OPC-UA PubSub doesn't really define the transport technology (different possible implementations). You can use eth broadcast or upd multicast, then there is MQTT 
• H: MQTT has reference implementation
• JP: for each of them, 
• H: that is the intel prototcol
• JP: yes, Ethernet, UDP, MQTT all not ok. Something like that is needed. But the big difference between the drones and the robots is that IA has strict TSN, we must really guaranteed latency, in Drones not strict TSN requirement
• H: ok
• JP: we are still in the same direction, problem 5 and 6
• H: need to investigate on item 5
• Jan: done some investigation on item 6. It should be possible to implement secure OPC-UA PubSub and we know how. On problem 5, we need to investigate further
• JP: would love to have a prototype keep-alive to play around.
• S: need report
• JP: PLC, we tried 1 year to integrate ProviewR in SlapOS, it uses for real by SSAB, it has fantastic documentation, but it is not modular, it is extremely difficult to modify build or split the different parts, what we did another PLC called Beremiz, interesting feature is support of PLC-OPEN, it is XML representation of I forgot the number. The Beremiz took the standard PDF 61131-1 and processed automatically standard document, generated the code PLC from translating every part of PDF into a programm. Then its completely modular, using Open-XML for data exchange. Then uses Matiec to compile instruction lists in the end. Used by three russian and one slowenian companies, guy lives in south of france and is ready to work for us to extend the system. Seems to have a long industry, so we are no trying this. ProviewR is impossible to build in different components. Doesn't work. 
• JP: Beremiz guy said, the instruction list it generates wil work with Beremiz and Codesys, but Codesys is not respecting standards for graphset and ladder. There is no complete implementation on the market, every vendor is just implementing it partly. Ivan is currently trying to use it
• Ivan: yes, using the IDE, run it in a runtime in another machine, so I'm able to read Modbus status (after a few hours). It's super-modular, compiles code using C, we can just run in a runtime in the cloud. 
• JP: supported Canopen standard, ready to implement OPC-UA inside Beremiz
• H: there is also PLC-Open Compagnion Specification
• Ivan: read it, working on specification
• JP: guy from Beremiz is specification maniac, specification to code generation machine, very compact code, to generate everything else. He has 14 years of experience in industrial automation
• H: sounds great, we have all the puzzle pieces
• JP: there will be need to properly handle real-time, need something to control strict real-time (using Xenomai?). We will use Preempt-RT
• Ivan: there is a cycletime in Beremiz, we have to handle.
• H: there are things in Linux to do
• JP: we gathered all parts, we need your help on OPC-UA keepalive transponder
• H: gives us a bit of time
•  

Minutes 01.02.2021 Discussion Fokus next steps

Participants: Frederik, Balakrishna (Fokus), Jean-Paul, Ivan, Sven (Nexedi)

Agenda:

1. Discussion of OPC-UA demonstrator

Discussion:

• Fred: Balakrishna will take over, Frederik goes to new project
• JP: requested SlapOS protocol documentation, used for 10 years, state is good enough, 40 pages, defines current state of SLAP protocal, with proposal for evolution, good enough to share with Fokus and prepare next step
• JP: we want to move from undefined/unclear protocol, to have an implementation independent protocol. We have mostly specified this. Next step: modernized protocol with OPC-UA implementation, so it would be very helpful if Fokus could help on this OPC-UA protocal version
• Ivan: sent email with summary, how to continue, from technical point of view, Fokus already did one API call with legacy way. Working, we can send data.
• Ivan: continue this proof-of-concept, continue working on new protocol with OPC-UA, jio and REST, idea is to replace our implementation on client and server side. Possible tasks for Fokus, and also to peer-review our document, need many remarks, to make it more clear and better to read.
• Ivan: need to show new SlapOS implementation proof-of-concept
• Bala: presentation, developed SlapOS documentation, wondering what needs to be modified, improved, changes required?
• Ivan: It was first pseudo implementation of one API call, before continuing, you need to understand our document
• Ivan: I will prepare real example in the meantime, it has to implement something on OSIE
• JP: having a 3rd party implementation of Slap Protocol would make us very happy. Two from Nexedi, 3rd from Fokus would be superhelpful. 
• JP: (showing SimpleRAN website) - describes how to use 5G networks only with software/source code, technology to turn PC into 5G broadband station. Regarding standard (...) we plan to use the Slap protocol for 5G orchestration of big networks. So we are moving into direction of a standard and it would be great if Fokus could make an independent implementation. SimpleRan already involving quite a few companies in DE. A lot of things can be done with same protocal. There is no protocol or standard for Kubernetes, it is not a standard specification, isn't it? I think having a specification and standard can have a very high value.
• JP: We need to have a good standard of orchestration, deployment control in data center, factory and 5g network, and this is what Slap does
• JP: can I see implementation that was done? 
• Ivan: super simple, but using OPC-UA as a carrier
• Fred: OPC-UA client-server model for non-TSN use, if you go to use TSN sensor/actutator level, then TSN will become more interesting and we need to use PubSub. 
• JP: for non-real time client-server model, but for PubSub for real time (looking atimplmentation) What is UA model ?
• Fred: information model used on OPC-UA server, this defines, the data-space or "adress space" . Everything is a node (object-nodes, method-node, etc). We defined one node for every SlapOS verb. Then you have Node-types, a node-type represents a SlapOS slave. Every new slave connecting to master will have his own node.
• JP: data-space? Used by IDSA and Gaia-X, seems to be a hot topic for data sharing among industries. Do you know people using OPC-UA ?
• Fred: nothing to do with each other, ... provided by the library
• JP: library is a wrapper over Open-65...
• Fred: no, it's independent, this was a compromise, because the example provided was Python 2.7, and because we wanted to do as little wrapping as possible, this is the only library available, not maintained, because it will soon will be legacy, I would encourage you to move to Python 3. 
• JP: we are moving to Python3, current state most of SlapOS except ERP5 should be Python3 compatible. Moving ERP5 to Python3 has almost already failed, it will take one more year at least. Moving to Python3 can be so difficult, for real industrial projects, many companies switch to GO, because ... same price. Python3 is creating huge difficulties but only on very big software. It is a topic, that make me very irritated. Broke pickle format, transparent cast of strings to bytestreams
• Fred: doesn't matter too much, we can use C, Python stuff only concerns high level components. We use Open-62... which also has PubSub, quite interesting, because this is the OPC model dedicated for TSN. 
• JP: for SlapOS no need, but we want to use it for real-time communication. I wanted to use PubSub between TLC and sensor/actuators, overall architecture will be PubSub between TLC and components and the other model for the rest
• JP: Ok so in this proof of concept, the info model in XML the kind of OPC-UA profile definition, so it's implementation indepdenent
• Fred: yes you can load it into any OPC-UA server
• Ivan: tried it, works
• JP: what I forsee, is that in future implementation, we will use a lot of UA-object type and the method type being used here, UA-method. Currently (sharing screen with specification document): we first tried to define "Compute Node" and "Compute Node Status" etc we try to define "Object types" with some schema.
• JP: every method has a verb passing parameters
• JP: all methods will have same name eg post with an object, so we move from verb-based to content-based API. If I translate this into OPC-UA, we are going to define a lot of UA-object type but in terms of UA-method there will maybe only be 4 methods (POST, PUT, DELETE, GET). Can it be generic?
• Fred: I would suggest that on the server side we parse on the delivered content and act on this, but not sure if OPC-UA method can have a variable amount of fields and if we need to fill the remaining with dummies, need to check
• JP: now for example in UA-method (...) 
• Fred: you define the method and it can have multiple arguments
• JP: where are they defined?
• Fred: don't know, but createNode only takes one argument = Name of the node. yes, there is only "node-name"
• JP: how is UA Method linked?... ah they use XML to describe a tree, I see. Question is whether it is possible for this UA-Variable with parent node XYZ, can the parent have multiple children?
• Fred: yes
• Fred: so it can have either those arguments or many arguments, but they need to be predefined, that is part of information model, that means for the POST method, we define 10 input argument,s because one verb needs 10 arguments, but another verb only needs 2, question is how to fill remaining ones.
• JP: current verb based SlapOS fits easily into OPC-UA model. Next step very useful to know how we can make the POST api fit into OPC-UA model. We should make a POC not with a single method name, but with two method names. 
• Ivan: POST and PUT implementations. 
• JP: imagine 2 different POST types
• Fred: interesting part will be different amounts of arguments
• Fred: you can pass Node-ids referencing existing nodes on the server
• JP: mapping of UA-Object type looks so much like JSON schema. Next step is to know how we can map GET/PUT/POST/DELETE
• Ivan: for me quite useful and quite important.
• JP: maybe possible maybe not, we should start here.
• Fred: we will start with this, using the same mockup, with Python 2.7, we can branch off from this repo. From now it is a mockup, and only mockup, just to be sure we mutually understand for communication
• JP: Ivan spend 3 months reading SlapOS code, internally at Nexedi he asked the people who are best in SlapOS to review it and make sure it's complete, if you could read it and say "we don't understand, etc." it will help us improve the document and is a way to provide documentation
• Fred: we can base new mock-up on this document?
• JP: for now only focus on legacy API, because it is complete. Then we intend to move to new API but this will take one more month to stabilize ideas. Legacy = reality it works, Future = dream, still need to design it so that we have proper OPC-UA implementation
• Ivan: to be clear: we need proof with new API.
• JP: from understanding use legacy, from future perspective use the new one. Very useful for me, the prototype to better understand OPC-UA. 
• Fred: give feedback. New mockup POST/GET ? And content parsing on the server side
• JP: yes
• Fred: on server we identify action to be run.
• JP: there is a content-type which is unique
• Fred: no need to identify action based on arguments, there is a unique id or something what to do.
• JP: what is important is, that the structure of the arguments represents (...) on the server. In the code, when I see UA-Object type, etc this kind of data structure is not so far from the schema's we pass as parameters. If we call post we tell server to add object with this schema
• Bala: python or C
• JP: only need PubSub for interfacing virtual PLC with actuators. Use of PubSub and client-server model will independent
• Bala: Python OPC-UA implementation, open-654... reviewed by OPC-foundation. In terms of new implementation...
• JP: for now for mockup, no need to change library, but of course on Nexedi side, we will be using open-6... 
• JP: for now we don't need only need to know how API will look like. Moving to content based API everywhere, gives a uniform programming model, Better to validate this will work with OPC UA
• Fred: project plan tasks and deliverables. 
• JP: in terms of deliverables we have to stick to deliverables. In terms of deadlines I want to extend project to stick deliverables. Ivan and Sven asked same question. I replied if we don't have Slap protocol specification we only delivery useful things. Now we have something. I think we can follow deliverables.
• (...)
• JP: helpful how to translate things to reality

Minutes 13.11.2020 Discussion on SlapOS and OPC-UA Integration

Participants: Frederik, Balakrishna (Fokus), Jean-Paul, Ivan, Sven (Nexedi)

Agenda:

1. Discussion of OPC-UA demonstrator

Discussion:

• (Ivan) (quick presentation of OSIE components and gitlab repo with SlapOS example)
• (Frederik) on API table include columns for upload and download, how does this work? Is it request-response ? Is the master supposed to reply (because no verb currently has a response from the master)?
• (Ivan) Master does reply but not everywhere, however never out of own initiative. System is a pull, not a push system.
• (Frederik) focus on supplySupply method which sends a computer id and string. What about special case of master talking to master?
• (Ivan) needs more discussion, for now, let's stay with simple example
• (Frederik) you want to deploy OPC-UA on SlapOS for Slave-Master communication and TSN?
• (Ivan) yes, we're still discussing whether ERP5 itself should directly understand OPC-UA or whether there should be a middle broker
• (Frederik) for lowlevel stuff, use open62541, for SlapOS better go with other solution, no need for PubSub, request-response is ok with open65421
• (Frederik) we should dicuss an OPC-UA client deploying slave nodes and an OPC-UA server running the master node, that way framework can just work as SlapOS using OPC-UA clients to make request to master nodes and having a OPC-UA server responding back to slaves.
• (Jean-Paul) SlapOS Master should act as OPC-UA server. 
• (Frederik) yes
• (Jean-Paul) currently SlapOS Master is acting as HTTP server, we want it to act as OPC-UA server. Node is acting as HTTP client, should now act as OPC-UA client. In relation between client and server, we have interested whether we want to use remote RPC approach or PubSub approach of OPC-UA
• (Frederik) PubSub whereever TSN will come to use, SlapOS for ochestration and management is neither time critical, nor cyclic, nor periodic traffic. If this is the case, PubSub makes less sense.
• (Jean-Paul) we're considering extending client-server with real-time notifications, currently, polling make our system very resilient, but also slow.
• (Frederik) OPC-UA would also support unsoliticited notifications
• (Jean-Paul) we'll follow your advice. One day we would like to wake up a SlapOS node to start polling faster than current sleep phase
• (Frederik) can be done with notifications, part of OPC-UA, can invoke clients
• (Jean-Paul) ok
• (Frederik) we suggest starting by building an information module. supplySupply verb, will be more or less identical to other verbs, we'll start with computer id and string
• (Ivan) will send links to OPC-UA Python library he has been using.
• (Frederik) other questions? Else starting to work on a prototype and keep in close contact
• (All) nope

Minutes 27.10.2020 Discussion SSAB on working with ProviewR

Participants: Carl-Hendrik, Claes, Kristoffer (SSAB), Jean-Paul, Ivan, Esteban, Sven (Nexedi)

Agenda:

1. How to get help on ProviewR

Discussion

• (Esteban) showed where he was stuck
• (Jean-Paul) walked through our last published article, so SSAB understands what we are trying to do with Godot and then with the sorting machine - https://www.nexedi.com/blog/P-OSIE-Blog.ProviewR.Godot
• (Jean-Paul) issues with building ProviewR - https://www.nexedi.com/blog/P-OSIE-Blog.ProviewR.Build
• (Jean-Paul) different things interesting for you, for example, shrinking ProviewR distribution on SlapOS as much as possible, enable/expand original ProviewR tests, complete ASan assisted cleanup of memory access, adopt industry standard interfaces between ProviewR components (eg use ProviewR PLC engine with other IDEs), somewhat verify runtime code with Frama-C and make build process independent from Xorg and add OPC-UA
• (Carl-Hendrik) OPC-UA is also a topic for SSAB, we can support on technical specification
• (Carl-Hendrik) For debugging, maybe best to connect Esteban directly with Kristoffer, then it is easier
   

Minutes 15.10.2020 Fokus Discussion on Slap protocol OPC-UA Demonstrator

Discussion

• (Ivan) we are learning what is needed
• (Ivan) he will be our contact, so far not much done
• (Frederik) not specialist in OPC-UA, but they have resources he can rely on
• (Ivan) will lead 1-2 years to have a Slap protocal with OPC-UA
• (Frederik) Q:proof of concept based on one of the APIs in the key
• (Ivan) define 1 API, very simple, we do a mockup with Python and then we see how it goes
• (Frederik) Q: what is SlapOS, core and slave nodes, productive employment
• (Frederik) Q: and you will use ProviewR on SlapOS
• (Frederik) Q: slave node will run on Olimex Node
• (Ivan) Use powerful machine with Master ProviewR and then just pass configuration to Olimex nodes
• (Frederik) Q: what do you have running
• (Ivan) ProviewR is available, but it uses a lot of shared resources and tmp/ and root access, won't work on server with 200 instances, hardcoded local interfaces while SlapOS uses local interface, partition cannot have root privileges. So one cannot kill the other. Issues, one of the nexedi employees fixed stuff, but tough
• (Frederik) Q: programming language
• (Frederik) Q: need Slapos, can we have a master or slave
• (Ivan) avoid going deep into Master and Node, so we do mockups. 
• (Frederik) Q: who provides mockup?
• (Ivan) I will do faster, rather you finding out how everything works. 
• (Frederik) Provide Mockup, then we can start implementing. More details about protocol example to implement, XML
• (Frederik) how to provide ane xample, network example, wireshark, no need because we need to define a new protocol
• (Ivan) no need to dive deep, because not happy with the mess
• (Frederik) Partner institute - machine park, assembly hall, research Fraunhofer APK.  
• (Sven) We write, they validate
• (Ivan) 1-2 weeks
• (Frederik) WP2 

Minutes 02.10.2020 M6 Deliverables Discussion and next steps

xxx

Minutes 24.08.2020 LTX Discussion on Implementation of software-defined networking in Linux

Participants: Heinz, Kurt, Thomas (Linutronix), Jean-Paul, Ivan, Johan, Sven (Nexedi)

Agenda:

1. Discuss and agree on way of implementing software defined TSN in the Linux kernel
2.Summary Jean-Paul

Discussion:

• JP: objectives real system and marketing (claim eg 35 μs precision)
• TG: more a challenge of latency
• JP: in Johan's setup, ethernet controller has two queues, one with strict priority, just wanted to use this and send critical packets on this queue
• TG: doesn't change the topic of latency and ensuring that packet goes out on time, challenge of what else system is doing and jitter
• JP: in theory, latency 30-40μs, jitter 20μs but in practice 200μs will still work for many cases and easy to achieve with time-stamping, computing in advance
• JP: main issue security, heard "security is never done with Linux", so 1 network to control, 1 network for safety
• TG: always need fallback, easiest is go into safety state and wait for system to recover. Easy for robots = stop. More complicated in other areas, what is "safety-stop" equivalent for self-driving cars
• TG: safety is question of design. Low complexity may be ok with one network and "all stop" and resync if latency is known. More complex cases need two networks
• TG: can also be multiple voters computing same decision, then decide 2 out of 2, or 2 out of 3, safety is probability of failure, so if 2 out of 3 works in 99,999999999999% cases, you can get away with it
• JP: for OSIE safety is single network, if something out of sync, all stops, using negative logic, always send packets if one stops receiving, everybody stops. So 1 network, no msg, all stop
• JP: case of Johan 2 motors, with time stamping great precision, but latency is an issue. If one of 2 motors has defect, it takes 200μs by microcontroller to detect and stop other motor, can this be improved?
• TG: SOS messages should not make a server roundtrip, they should go to all devices
• JP: have feeling hard to get below 100μs
• TG: period of full communication determines scenario: 1) network speed, 2) reaction time of receiver and computation time to dispatch next data
• TG: keep in mind, you also can't send when you want, you need to respect slots, else you break the system
• JP: TSN allows to define different slots for different types of communication
• TG: SOS can also have a fallback wire, with this reaction is close to 0, there are different ways of mitigation
• TG: Focus should be to prove we can do that, could already be sufficient for many applications, if more needed it's an issue for controlbus. All busses have well defined touring times + extra measures for critical situations
• TG: Intersting to prove ths a software TSN implementation can achieve X and X depends on Y and Z, then you can get to the limitations and user can decide. There is no generic solution to all use cases
• JP: need to prove software TSN and low level sync, else industry will not consider us serious, check boxes for TSN and OPC-UA
• JP: Question: 802.11.QAV, 2 WLAN, use as queues, manage two networks
• TG: wrong standard it's quality of service, doesn't fullfill gurantee of periodic behavior, dedicated time slots not possible
• JP: interesting 2 WLAN, give priority to one and implement time slots would split problem of routing and real-time with different adress ranges
• TG: still doesn't gurantee time slots, Linux supports notion of fully periodic timeslots
• JP: software or hardware?
• TG: both, less accuracy for software, question will be what hardware support possible and usable. Software will be interesting challenge the more complex it gets, also network controllers have interesting (mis)behaviors and are optimized for throughput, not for latency, so they might aggregate traffic before sending, which we need to disable. We need to chose carefully what components we chose.
• JP: ability to get low latency, high precision using generic hardware, eg OCP server + xyz. Afraid of expensive TSN compliant hardware
• TG: capable and affordable devices exist, not out of reach. Interesting part is device side and whether you can use software controller implementation.
• JP: new board from Olimex is better for that, so higher priority to to on microcontroller, objective is to use Olimex instead of Wago
• TG: start from this side an ignore server in the beginning
• JP: switch can be a problem, no idea of TSN support
• TG: switches exist, mostly meant for networks where not everyone agrees on schedule and act as gatekeeper, switch knows and manages all
• JP: maybe get support from Accton
• TG: not now, first we solve on our small setup. Regular switch will be fine. 
• JP: we will send specs for target setup, LTX is also working on OPC-UA, so consider OSIE as test user for your current implementation, we can start with 10ms and improve. I'd love to reach something better than the 35μs
• TG: ok, period of communication and time for computation and send feedback determine the limits of what is achieveable with safety on one network
• JP: routing protocol seems incompatible with TSN time slot optimization, as routing finds best path without guarantees. Best design seems two different adress spaces, either way, we'll still try to publish a paper
• TG: needs investigation

Summary Jean-Paul

• Linutronix => Secruity, alternative Linux, software defined TSN
• Safety = if out of sync: stop, if no message: stop
• TSN = time slots for device or server to send messages, period of schedule = round trip time
  • feedback => send next message => display feedback
  • special messages could be recived by server and other device
  • 100 μs network
• Profinet by Siemens
  • well defined round trip times
  • extra for critical requirements
  • extra mechanism to handle exceptional events
• "Safety Net" (fallback) => Safe state depend on what you are controlling, complex system eg self-driving cars may have different safe state than "stop"
• Software TSN main network
  • time slots
  • different devices on network
  • can communicate
  • general purpose time slots
  • in case of any violation of latencies or constraints, we should "just shut down"
  • if safety requires more, then need 2nd option to have fallback control

Minutes 18.06.2020 SSAB/ProviewR Briefing

Participants: Carl-Henrik (SSAB) Jean-Paul, Ivan, Sven

Agenda:

1. Discussing possibilities to contribute/help development of ProviewR

• (relocation) push in the upstream our changes that make ProviewR relocatable to any directory ;

• (build) shrink ProviewR distribution so that the runtime can be deployed with fewer dependencies ;
• (build) make the build process independent from Xorg ;
• (build) simplify the build process by using a single tool or using a modern tool (to be defined) ;
• (security) apply ASan to ProviewR code and sanitize memory access ;
• (security) verify runtime code with source analysis tool such as Frama-C ;
• (standard) extend ProviewR with standard interfaces for certain components so that it is easier to connect to third-party tools (eg. IDE) ;
• (standard) support OPC-UA in ProviewR ;
• (networking) support TSN in ProviewR ;
• (programming) support Cython / Python programming in ProviewR (already done maybe).

Due to Covid-19 SSAB is currently only working on 60% and summer holidays are coming up, so from their side, resources are very constrained. Nexedi has budget available to finance development of improvement of ProviewR.

Discussion

Build

• Self-contained build, Dmitry will analyse and suggest a tool, review by Romain. Requires to really understand ProviewR and why its current form is required for actual industrial usage.
• Trying to understand why Qcom is used for distributed system and forwarding messages from node to node = what is better in Qcom than in comparable software?
• We plan to use ProviewR on central server controlling remote device Modbus over OPC-UA, we will try to run 1000 ProviewR processes on a OPC server on bare metal without VM/containers, requires mastering the build and being able to configurate parameters to not depend on things like /tmp directories
• We will make pull request, SSAB should give proper feedback so we can improve

OPC-UA

• SSAB are looking into this themselves, will come back to use in 2-3 weeks after speaking with their developer staff

Security

• Dmitry found many security issues and would like to fix them using automated tools to sanitize memory access
• JP / Dmitry suggested using Frama-C to verify C code properties for ProviewR runtime
• Logic behind Frama-C usage is that it's used by the French CAA nuclear energy center and their code verification team has high science abilities / well introduced to research financing. We would like to use Frama-C (written in CAML) to give a security showcase and get interest for ProviewR on industrial and defense side
• Industry 4.0 wants to connect MES + Fieldbus + Internet because how to sync production over multiple sites if no access to internet, we would like to prove security of PLC runtime helping to make the Fieldbus more open; SSAB interesting, because actually looking in other directions and how to make their network more secure and lockdown subnets from internet. It's more or less large industrial standard to have no access to internet, becuase when hardware (Scada etc) was never build for security, PLC vendors (Siemens, Schneider) now try to use two networks - one for the fieldbus, the second one for (Azure) Edge Computing = box with access to Fieldbus and Internet = not secure, In general it is an interesting research topic, because Industrial Automotion 4.0 requires connectivity 

TSN

• might be not very interesting in reality, because overprovisioning of network (eg with 10GB) is so cheap today, but we still want to prove that we can do it on standard cables
• achieve 35ms jitter/latency to sync motors to show what is possible with only open source software

Cython

• Python language which allows to run with same speed as C++ and work on multicore processors, more and more used for embedded systems at Nexedi
• SSAB is also working on an interface with Python but they don't run any Python code inside ProviewR - exachange status in either way, so we don't do duplicate work.

Questions

• Workbench allows to model a PLC and compile a runtime, upload through SSH and execute
  • Is there a text based serial format to exchange model with precompiled runtime ?
  • Textfile defines logic of PLC, upload to already compiled runtime and execute possible
  • SSAB: possible to upload a textfile and turn off some values, but not possible to replace the whole thing
  • Always require to build through compliation ?
  • SSAB for change in values not required to recompile, for change in logic or database access you need to recompile, but depends on use of ProviewR. A hot swap is possible, swap a running object, but SSAB not using it

Next steps

• Share info in 2-3 weeks on availability for development in France/Sweden financed by Nexedi
• Dmitry is building a virtual factory to be run by ProviewR
• Next call in 2-3 weeks

Minutes 17.06.2020 Fraunhofer Action Plan

Participants: Varun, Frederik, Alex (Linutronix), Ivan, Sven

Agenda:

1. Software Development Capabilities
2. Discuss contribution of Fraunhofer Fokus

Software Development Capabilities

• Development mostly related to generic communication technology. Varun is leading a developer team working on real-time edge node, mostly scheduling of real-time tasks (RFC XXX), Frederik's team is working on a library to allow apps to schedule tasks on time-sensitive networks (XXX I 2010). Fokus edge nodes, time sensitive network over fiber optics and 5G.
• Main issue relating to OSIE is that at Fokus not everything is open source, since we do also industrial prototypes. 
• Core development in real-time extensions and pgp support, interested in low level

Possible contributions

• Time sensistive networking is most interesting, testing and evaluating how good determinism is on an architecture that is purely open source. They have existing proprietary systems but there is no comparison with a fully open source approach => we need to verify to not duplicate work with LTX
• Worked with CodeSys, not heard of ProviewR before OSIE, not sure whether integratable
• Work with OPC-UA extensively since it's the de-facto standard. 

=> NXD needs to define SLAP protocol, afterwards, FKS could help us to define protocol for service discovery and scheduling based on OPC-UA. FKS has the experience in this field to guide the development and help us make all OSIE components become OPC-UA compliant (most notably ERP5 and ProviewR)
=> FKS could validate the work done by Linutronix. 

Minutes 10.06.2020 Linutronix Action Plan

Participants: Heinz, Kurt (Linutronix), Ivan, Johan, Sven

Agenda:

1. Discuss approach suggested by Jean-Paul and Juliusz

Approach

Linutronix: the approach could work in principle, but they have to discuss it in much more detail. During the Picasso project, Linutronix evaluated several public networks and found that strict timing constraints may work in closed domains, but once routers are involved, delays of several milliseconds were encountered. Nexedi: We will have to measure carefully the hardware we are testing and we should consider also adding a router to our setup. The setup should be reproduced on all locations, so we can work in parallel. Also, we should make sure that new OMX boards run the mainline Linux version 5.3 or 5.4, because it will make life much easier.

Linutronix will come back with more detailed feedback shorty. Nexedi will give access to Johan's work, so it can be included in the feedback, Ivan will check which Debian is running on OMX boards.

Additional notes:

• Nexedi: Eventually, we will probably have a similar approach to Wendelin where we test and patch external (public) libraries for memory consumption to ensure the work in Wendelin. For OSIE this would mean testing and classifying hardware based on the measured delays (traffic light classification) and providing a list of OSIE "certified" hardware components. 

• Nexedi: Since the guaranteed route is manually configured per the recommendation of Juliusz, Sven wondered how this route can be maintained/optimized, eg someone physically stepping on a cable. If critical information is sent through this route, there needs to be a safety circuit.

Minutes 28.05.2020 Kickoff Call

Participants: Alex, Frederik, Varun (Fokus), Heinz, Kurt (Linutronix), Tsvetan (Olimex), Jean-Paul, Ivan, Sven (Nexedi)

Agenda

1. Introduction - tell us who you are
2. Using the Project Section and Forum
3. Current Status - everyone should tell what they have done so far
4. Additional Points

Introduction

Project Section/Forum

Current Status

Nexedi :

• 2 FTE working on the project working on :
• integration of ProviewR (open source alternative to Codesys) - currently possible to recompile and redeploy, mass provisioning of PLCs is possible in theory - 12 months to actually work
• deploying real-time Linux using Preempt_RT patch on Olimex board, trying to get an idea of the jitter (currently < 100ms) to educate ourselves and learn to use plain generic hardware to eventually achieve vendor independent TSN
• deploying data lake (in another project) with sensor data collection which needs to be replicated in OSIE
• developing low-cost 3d printed boxes for Olimex boards based on Wago equipment

Alex: 12 months to work, what is missing ? => cheap hardware for espressif ESP32 microcontroller, needed for presicion time control as nobody really provides TSN. It is possible as we already deploy 4G/5G networks over ethernet between baseband unit and remote radio heads. Principle of overprovisioning of bandwith in order to measure TSN jitter.

Linutronix:

• still a lot of work for mainlining Preempt_RT into Linux Kernel, next support of PTP (precision time protocol)

JP: using Edgecore switch from Accton in Taiwan, so open source PTP/time stamped hardware possible

Olimex:

• working on industrial soldering robot, required to automate production, currently doing classification tests to try and make robot intelligent (evaluate work after soldering). Robot is using Olimex electronic hardware (CNC) running Linux

Fraunhofer Fokus:

• 1/2 FTE available (including Frederik and Varun), currently also having a student working with Codesys which might be useful

Other points:

• Alex: Eureka Ethics survey to fill out
• JP: building a production line POC for cherry sorting during next 6-8 months using all components
• Alex: Travel restrictions at least until end of the year
• JP: Experience of Fraunhofer Fokus with PTP ? => 5G Link endpoint synchronisation and IEEE TSN group is using PTP in 802.1AS-Rev = the generalized (and more simplified) variant of PTP (gPTP), mostly to support Layer 2 (which makes sense for a TSN standard) and peer delays
• (follow-up) Heinz: open source PLC for distributed systems, we could also have a look at 4DIAC (based on eclipse), maybe it is also usable for experimenting
• (follow-up) Heinz: looked at ProviewR a long time ago, difficult to make compatible with Codesys, but considering that the Codesys Editor is free to use and clients have enquired why there wasn't an open source solution configurable with the free Codesys editor, we should maybe consider this as part of OSIE to have an argument for selling
• (follow-up) Heinz: AccessTSN will come to an end at the end of June. The main purpose was to establish the basic support for TSN (with a focus on endpoints and switched endpoints) in mainline Linux, which was done successfully. Support for Intel i910 is there, NXP is following, and Kurt is working on upstreaming support for a Hirschmann device (switched endpoint). Missing is the one or the other small point, e.g. with PTP adaption to the latest release of the spec. A big missing point is the tooling – how to bring a configuration over a number of switches to the endpoint and vice versa. And how do I trace the configuration in case of misbehaviour.