Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Main Loop Monitoring - Debug help #17

Open
1 of 3 tasks
RobertK66 opened this issue Apr 9, 2022 · 6 comments
Open
1 of 3 tasks

Main Loop Monitoring - Debug help #17

RobertK66 opened this issue Apr 9, 2022 · 6 comments
Assignees
Labels
bug Something isn't working feature New feature needed for CLIMB mission

Comments

@RobertK66
Copy link
Owner

RobertK66 commented Apr 9, 2022

  • Add meassurments to detect mainloop anomalies regarding to runtime of module main routines.
  • move the 'LaststartedModule' debug featur to the mainloop. (Expand it with 'IRQ marker'
  • add (debug) functionality with global availability of raw/string event.

Meassure runtime per module.
include 'current running' module to monitor structure
make and persist events absout treshholds violations

  • single module out of 'optimal runtime'
  • single module out of 'allowed runtime' -> disable calling of this module
  • main loop execution out of 'optimal' runtime
  • main loop execution out of 'alowed' runtime -> ??? / extra thing needed or is Hardware Watchdog good enough in this
  • (try to) make usefull event with 'current module nr' and 'soft reset' with hard faults and other fatal IRQs.

make 'module enabled/disabled' possible. Switch on/off (with 'reInit' or without !? switch off with DeInit !?)

@RobertK66 RobertK66 added the feature New feature needed for CLIMB mission label Apr 9, 2022
@kodsurf kodsurf changed the title Main Loop Monitoring ISSUE-17: Main Loop Monitoring Sep 14, 2022
@kodsurf
Copy link
Collaborator

kodsurf commented Sep 14, 2022

I added header file that stores global variable

LAST_STARTED_MODULE

Included that header to all files.

For every function writen I MANUALLY set up unique index corresponding to this function.

image

When any function withing the module starts - it sets LAST_STARTED_MODULE to corresponding index

example : index 11 means that it is l4_thruster_main()
1111 -is void thr_execute_sequence(int procedure_id) function within l4_thruster.c

This way when HARD FAULT OCCURES global variable with index of last started function will be saved and visible for debug


EXAMPLE :

The goal of implementing this watchdog is to investigate what causes failure when executing sequences in feature/thruster-fire branch (ISSUE-25) #25

Current issue :
When command "8 0" is sent. THR_HARDCODED_SEQUENCE[0] is executed (test sequence of function pointers )

After "stage 6" SOMETIMES failure occures.

image
image
image

5a38f45

@kodsurf
Copy link
Collaborator

kodsurf commented Sep 14, 2022

I discovered that failure is caused by thr_execute_sequence() function within l4_thruster.c

Thus I added additional "breakpoints" to find out where exactly error is occuring

image

I launched sequence with "8 0" command again and waited for failure.

First attempt resulted in no error occuring (sometimes work, sometimes not)
image

After second launch of sequence

image

With an error after
image

thus

image

Error occured by executing function at specific pointer (probably wrong pointer ?)

@RobertK66 Please look through this procces of investigation

@kodsurf
Copy link
Collaborator

kodsurf commented Sep 14, 2022

I were able to SUCCESFULLY execute "8 0" ANY NUMBER OF TIMES without an error.

Error occures ONLY AFTER another sequence was executed

for example "8 1" - after that "8 0" will trigger an error at "stage 6"

image
image

So it failed when trying to execute a function of THR_HARDCODED_SEQUENCE[proccedure_id].sequence[8].function

In ORIGINAL sequence this is thr_wait() function. And its defination and argv are absolutely correct
image

BUT !!!!!!!!!!!!!!!!!!!!!!!!

I noticed that when I programmed sequences in l4_init() !!!!!!!!!!!!!!!!

I used temporal variable
image

NOW IMPORTANT BUG !!! - Length of original "0" th sequence is 12

length of sequence "0" is 7.

Execution of "0"th sequence after execution of "2" sequence causes failure due to wrong function pointer !!!!

Why is that ?!?!?!

@kodsurf
Copy link
Collaborator

kodsurf commented Sep 14, 2022

image

I think problem is in this defination ...

idea behind thr_hardcoded_sequences_t THR_HARDCODED_SEQUENCES[5] is to store ARRAY of a type thr_sequences_t

image

is meant to be AN ARRAY of function pointers and arguments

and thr_sequence_t is a typedef of single element in this array

image
probably here is an error in defination ????


I want to save array of structures as an element of another structure ! How can I do it ?
image

@RobertK66
Copy link
Owner Author

Ok, lets keep this as a 'feature issue' to implement the 'current running module for everybody in a basic mainloop implementation.
I also plan to include this and the generic string / raw event somewhere in the (ado?) module base with this.....

@RobertK66 RobertK66 changed the title ISSUE-17: Main Loop Monitoring Main Loop Monitoring - Debug help Nov 14, 2022
@RobertK66 RobertK66 self-assigned this Nov 18, 2022
@RobertK66 RobertK66 mentioned this issue Jan 16, 2023
6 tasks
@RobertK66
Copy link
Owner Author

I think I had one instance of 'Hardfault' triggered by the current calculate Runtime method in the mainloop ->
this is desasterous and I really think now to remove this 'complicated code' from mainloop ....
if kept, how to test for all possible timing and math problems here ..... ???

@RobertK66 RobertK66 added the bug Something isn't working label Nov 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working feature New feature needed for CLIMB mission
Projects
None yet
Development

No branches or pull requests

2 participants