;; This buffer is for notes you don’t want to save, and for Lisp ;; evaluation. If you want to create a file, visit that file with C-x ;; C-f, then enter the text in that file’s own buffer.
#-*- mode: org -*-
Sergey Levine and Pieter Abbeel’s new papers come to mind. Especially: TRPO, Guided Policy Search, …
Kevin Down et al. Compare with Bayes Opt approaches.
Benjamin Recht et al.
Include in the paper an Elastic Net path using LARS that makes more sense.
with basis function parameters that illustrate the importance of the learned movement primitive.
The algorithm selects very different parameters (even the number!) depending on the initial starting widths, heights, etc.
Include plot of how the algorithms remove some of the variables, i.e. 87-80-70-70-… on average
Added responses to all the reviewers’ comments Added a summary of the major points as bullet points
Why don’t we learn in Cartesian/task space? I argue that inverse kinematics would be an artifically introduced burden on top of the original (difficult) problemClarify that demonstrations are using kinesthetic teach-inModify Figure 1 and add cuts of demonstrations Figure 1 should have a clear background, also including cuts from demonstrations is a good idea. Revise Lasso and Elastic Net motivationMotivate/compare more to previous MP methods e.g. DMPs, ProMPs Remake Figure 2 I zoomed in on a new training plot with 20 demonstrations Here the emergence of the new parameters was more visibleClarify the use of ranking the parameters (subsection C)Add maybe plots of the ranked features?Discuss computational complexityCheck python code for derivatives Complexity of grad. eval + residuals (as function eval.) Include in appendix?Check python code for speed BFGS seems to be taking longer than the multi-Task elastic Net?Experiments based on parameter size would be enlightening Averaging would let me guess the comp. timeInvestigate Multi-task case In the normal Lasso, time-complexity is O(N*p^2). I suspect it should be multiplied by $d$, the degreeCompare the proposed methods LSDP and cLSDP in terms of computational complexity, compare also to DMPs Discuss the joints in Figure 3 In caption as well as in the scriptInclude smoothened ball positions in Figure 4Discuss table in more detailExplain the columnsDemonstrate why sparsity is importantExplain cLSDP fitting with more/less demonstrations In particular, how does having higher residuals relate to the quality of the solution?Discuss the ‘quality’ of the demonstrations, Are they important for the actual execution also?Discuss the generalization abilities of the method What are the limitations of the approach? The number of demonstrations would I think be the main bottleneck.The idea is to learn invariants of motion / parameterize the relative motion between the joints using a graph (graphical model?).
The improvement for RL is to find a reduced dim. latent space on which we can perform policy search/BO.
The policy that the graph generates is relative, i.e. it can be grounded by specifying an initial joint state and possibly also ball state.
Ideally the policy should also have low accelerations and ideally it should involve some sort of planning, e.g. it should know intrinsically the value of a certain distance from the ball.
Inverse Reinforcement Learning maybe? To learn the Value Function of the teacher?
Possibly from movement primitives and/or RL trajectories one could learn such a model.
Does the full planning problem with the two ball takeoff and racket impact models included result in a continuous trajectory? hybrid? polynomials?
If we can additionally solve for local feedback, then one does not need to optimize repeatedly 3rd/5th order
The approach must clearly be model-based as I do not want to have ‘N’
rollouts for each policy
As opposed to GPs or neural networks, we could first try local linear models as an extension of DMPs.
Model for the ball take-off, ball-racket impact or robot dynamics? Or all three? Couple the ball as part of the state as well?
We have to make sure that local policy generalizes to different ball positions and robot rest postures.
In high dimensional cases it seems important to do dimensionality reduction.
The initial posture of the robot corresponds to the ‘context’ of the problem and contextual bandits would learn a GP over the contexts as well.
Jan suggested performing PCA analysis first but in the end I proposed (Lasso-like) multi-task Elastic Net coupled with feature adaptation.
The recent BO survey should be of help.
UCT algorithm comes to mind
New serve() function in SL can perform an optimization whenever the movement is predicted to fail. The movement will then switch the optimization if a feasible solution is found.
- Added optim, detach flags
- Input q_act is parameter to initialize the optimization
- Rate of optim and other flags from player.cfg are replicated in serve.cfg
- Desired land location flag also imported from player.cfg
Check the python code for the derivatives
Added pytest in Python to check for derivatives
Generated Elastic Net path after training (coupled) LSDP parameters
We have rl-experiments/ repository where I include the REINFORCE algorithm that I emplemented
The MIPS MATLAB Policy Search and iLQG MATLAB code is also included.
using Jens Kober’s implementation
e.g. a Barrett WAM forward dynamics
Extend ILC with improved trajectories As we use ILC and update the models, the models can also be used to update the reference trajectories progressively.Relation to DDP?REINFORCE is not converging!
Calculates the log derivatives
Learning rate, policy structure, number of basis functions, obs. noise, etc.
Heiko 3d-print-ed an egg holder for me to try
I’m trying at the moment to record ball positions as well during the recordings.
The demonstrations are now shown in the new BORIS SL (sebastian/sl_xeno/sl_wam) with mass = 0.375 and mcm = 0.0! These are not optimal parameters at all! However after the racket holder change, this was the best setting without the PD control that I found.
With the egg-holder it is possible to teach, but the shoulder is as suspected difficult to move, even with the PD turned off. I was moving more the elbow so far.
PD should be switched off.
Extracting movement primitives with 5 examples so far. One can remove/add more examples and also basis functions to test on the robot.
Testing on the robot was tricky, as we need to make sure that during the kinesthetic teachin, the joint limits are never exceeded. It seems that the hardware limits are not the same as the software limits.
After teaching for the second time with the eggholder on the red side of the racket, the movement could be executed on the robot.
We should have a json file with additional settings that we can adjust.
MATLAB dumps the JSON file in an unreadable way. Opening in Python and formatting solves the problem.
Unit test added that checks if subsampling the sped-up signal restores it to the original movement.
Silly mistake found where qd_des, qdd_des were not updated!
Having multiple json files could easily solve this problem.
Testing different postures, different DMPs and different starting positions we see that the I.C changes the accelerations like crazy!
Jens idea to reduce the initial accelerations does not work unfortunately. Instead we can penalize the accelerations using Radial Basis Functions.
This actually generates smoothing splines in the nonparametric case and in the parametric case when we also put an l1 penalty on the parameters we get a (multitask) Elastic Net!
Testing the learned sparse RBFs on real robot is next.
There seems to be a bug in the C++ code.
The new logistic regression parameters must all be negative. I saved the projection matrices also in a serve_3d_ping_okan file. The matrices are not normalized (i.e., P[3,4] is not 1) but triangulation in c++ does not cause a problem.
Instead of running an external triangulation server, I can do the triangulation in a detached thread inside the table tennis library.
In Python I use SVD to solve for the rightmost singular vector, this is a linear triangulation method.
It is easy to debug the interface, I can stream the received 2d pixels and the triangulated 3d positions to a debug file online
I use now the num variable received from the vision server
If the header changes almost all the library recompiles
So far triangulation did not pose a problem
The linear triangulation method with SVD does not work in test. Instead least squares with P0[1:2,:] and P1[1:2,:] rows concatenated matrix gives the exact inverse in test.
This will obviously pose a problem online. Instead now I update the ball positions only if triangulation succeeds.
Checking the ball data in python, we observe that the balls disappear during the motion if the movement starts in the back, should be closer to the cameras.
The zeros received in the ball positions are removed and the balls are plotted via the timing with the joints. [We get absolute time both from joint positions and ball positions saved]
Yes it does!
Even if the ball-takeoff model is not known, one can use movement primitives for at least the first part of the movement.
I think when it is clear that the racket will not return the separated ball one can ‘fire’ the optimization.
Third order or fifth order (minimum jerk)? Which is better?
I would guess fifth order should be better in this case
The Desiderata for RL in robotics:
- Data efficiency
- no rollouts
- this implies model based or something in between (BayesOpt.?)
- Safety
- small and/or local perturbations
- model based is more promising
- Reward engineering
- unsupervised learning?
- reward shaping?
- building models on trajectories
- Parameterized policies
- starting overparameterized or underparam.
- relation to ILC as a result of optim.
The possibilities are:
- Minimum distance between the ball and the racket during movement
- Minimum distance between the ball and the robot court center
- Maximum y-velocity of the ball
or possibly a combination of the above.
Reward shaping comes to mind here.